Six-Ish Things I Learned This Week
Six-ish Things I Learned This Week | CW16 CY2020
This is the first edition of Six-ish Things I Learned This Week. Every week I come across many new things, be it in programming, networking or PowerPoint. In an effort to not forget all these things, this is my attempt to write them down in a concise and searchable way. Maybe you find something interesting as well.
This week is extremely Python-heavy as I haven’t been doing anything but writing Python code in the past seven days. Let’s see if this changes for the next edition. I am not ruling out any topics, but without further ado, here are a few cool things I (re)discovered or learned this week.
What does YAML stand for?
Let’s start this off with something slightly embarrassing. After dealing with and using YAML files for at least five years now, I could have sworn YAML stands for Yet Another Markup Language. During a virtual DevNet Express event this week, I discovered that it, according to the official yaml.org webpage, means YAML Ain’t Markup Language. You never stop learning four-letter acronyms, I guess.
Versioning Python packages with versioneer
Versioning a package or application can be a bit of a pain. Especially if you are releasing frequently and want your git commit hash to be part of a version string. This one is technically a cheat in this category as I have known about versioneer, a Python package to automate the process of versioning Python packages based on commits and tags in your version control system, for some time but never actually used it. So that I don’t forget it in the future, here is how you can add versioneer versioning to your Python package.
Install versioneer
$ pip install versioneer
First, add a [versioneer] section to your setup.cfg. This section will specify things like which VCS to use, what version style to use and where to put the version file in the source tree.
[versioneer]
VCS = git
style = pep440
versionfile_source = <path_to_your_package_root>/_version.py
tag_prefix = v
Next, we need to tell our setup.py about versioneer and that this is the version number that should be used. To do so, modify your setup.py to import versioneer and also use it as the
version string.
import versioneer
# [...] All other imports
setup(
version=versioneer.get_version(),
cmdclass=versioneer.get_cmdclass(),
# [...]
# Rest of your setup function follows here.
# Make sure to delete any other 'version'
# or cmdclass args.
)
Now, as a final step, we need to run versioneer install in the root directory of your project.
$ versioneer install
This will copy the versioneer.py script over, create a _version.py file in the source code directory (specified in the configuration by versionfile_source) and add a __version__ variable to your package’s __init__.py file. You can now call <name_of_your_package>.__version__ to access your version number.
This configuration will use the versioning style specified in PEP 440. In its most basic form, the version number will be the git tag. If you are 5 commits ahead of the last tag, this will be indicated by a +5 appended to the tag. To identify which commit you are on, the short commit hash will also be appended.
So the version number for a package built from commit 9cd166ecd1d25d1855801ebc05eaa6fa9d7779ae2705a8637a612b92c05ea5d7 that is 6 commits ahead of the last tag, which was 1.0, would be v1.0+9.cd166e. If your local directory additionally contains uncommitted files, this will be acknowledged by appending a .dirty. So the above version with additional uncommitted files would have the version string v1.0+9.cd166e.dirty.
Easily rate-limiting Flask apps
When building a REST API you might come to the point where you want to rate-limit access to your API. Rate limiting means that you put a limit on how many requests a user can make in a given timeframe and it’s super useful to keep your API from being flooded with requests that could lead to deteriorating performance or completely unresponsive endpoints.
If you are using Flask to build your API, there is a package to introduce these limits in a very convenient fashion. The Flask-Limiter package is an MIT-licensed Flask extension developed by @alisaifee on GitHub that lets you decorate Flask endpoints to have rate limits.
Let’s have a look at how to limit a Flask endpoint to 5 calls per second and 60 calls per minute.
First, install the package by running
$ pip install flask-limiter
With that done we can start rate-limiting our endpoints.
from flask import Flask, jsonify
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
app = Flask(__name__)
limiter = Limiter(app,
key_func=get_remote_address,
default_limits=["2 per minute", "1 per second"]
)
@app.route("/api/v1/random_number")
@limiter.limit("5/second;60/minute")
def api_random_number():
return jsonify({"number": 4})
As you can see, we specify the limits in a human-readable way with the @limiter.limit decorator. This example uses the remote address as the key to identify a user (provided by the get_remote_address function). You can also write your own function to, for example, limit the number of requests per API key or per user that might be stored in your session or elsewhere or make the number of requests a user has done persistent between restarts by using Redis or Memcached as a storage backend. Check the documentation here for all the capabilities this cool little package has.
Getting the trackingId from a failed request to the Webex Teams API when using the webexteamssdk
If I had to choose a Python package that has had the biggest impact on my (work) life in the past two years, it has to be Chris Lunsford’s incredible Python SDK for the Webex Teams REST API. Chris has done an outstanding job of making dealing with the API like acting upon local objects in Python.
As an example, instead of doing a POST request to /v0/messages with a JSON body containing a destination mail address and a message, as well as setting the authorization headers correctly, I can just do:
from webexteamssdk import WebexTeamsAPI
api = WebexTeamsAPI()
api.messages.create(toPersonEmail="mn@mail.com", markdown="**test message**")
Need more convincing how powerful this is? Let’s send a message to everyone that is a member of a specific space (or room). And let’s do that in 5 lines of code (I have omitted the two lines needed to import and instantiate the API object in the example below):
ROOM_ID = "..."
for membership in api.memberships.list(roomId=ROOM_ID):
api.messages.create(toPersonEmail=membership.personEmail, markdown="Thanks for being part of the space")
The SDK takes care of returning an iterable object containing memberships as well as dealing with pagination and Retry-After headers when we send too many requests at once.
Now, if a request fails, one thing you might get asked for by the Webex API support team is a trackingId so that they can trace your request through the system. This trackingId, however, is not immediately visible from the ApiError exception that gets raised by the webexteamssdk. You can access that trackingId by checking the raw response that led to the error:
from webexteamssdk import WebexTeamsAPI, ApiError
api = WebexTeamsAPI()
try:
api.messages.create(toPersonEmail="mn@mail.com", markdown="**test message**")
except ApiError as e:
tracking_id = e.response.json()['trackingId']
print(f"Tracking Id: {tracking_id}")
How slicing works in Python 3
Ever wondered how slicing, the ability in Python to select a range of indexes from a list (l[2:4]), works? There is no better way than finding out by implementing slicing on a custom class.
The beauty of Python is that everything is an object and you can override methods on your own classes to respond to built-in operators like +, - or, and this is what we want to do here, the index operator [0].
I recently wanted to abuse this system. See, for a little project, I didn’t want to implement list-like behavior but rather extract the start and end of a slice. What I was trying to achieve was the following:
>>> c = Custome()
>>> c[4]
Accessed key is 4
>>> c[0:4]
Start of the slice is 0, end of the slice is 4
Every operator in Python 3 is implemented by a magic method (a method beginning and ending with two underscores) that Python calls on your objects when invoking an operator or, for example, a cast operation. The most prominent of these that every Django developer knows is the __str__ function that returns the string representation of an object and allows the str() function to be called on an object.
The function used for acting on the [] operator is __getitem__. This method takes a key argument that can either be a literal key (think l[3] giving you 3 as the key) or a slice object. So we can act upon these differently.
class Custome:
def __getitem__(self, key):
if isinstance(key, slice):
print(f"Start of the slice is {key.start}, end of the slice is {key.stop}")
else:
print(f"Accessed key is {key}")
As you can see, I am checking if the key is an instance of the built-in slice object. If this is the case, I can access the key.start and key.stop attributes of that object to extract my slice range. For the record, there is also a key.step if you specify a step size (e.g. in l[1:5:2], 2 is the step size, similar to how the built-in range function works).
Pretty cool Python, pretty cool!
Validating data models with pydantic
The fact that Python is not statically typed is both a curse and a blessing. If a programming language is statically typed (examples would be Java or C++) that means that the type, be it int, string or a custom class, is known at compile time. Python is, by default, dynamically typed which means that you can declare a variable without specifying a type. The Python interpreter will infer the type of the variable based on its value. So when you type the code example below
>>> slogan = "Python is great"
>>> print(type(slogan))
<class 'str'>
Python will, in the background, infer that slogan is a string and has that type. Now, this is great because it means that you don’t have to worry about variable types, saving you a few keystrokes every time you declare a variable. What makes it tricky, however, is that, technically speaking, every time you write a function that has some parameters you have to check that that parameter is of the type you are expecting. To illustrate:
def do_something(str_input):
split = str_input.split()
return " ".join([s.capitalize() for s in split])
# Case number 1
print(do_something("this is a sentence"))
# Case number 2
print(do_something(2))
This piece of code works perfectly fine in case number 1 and capitalizes the first letter of every word in the passed sentence. Case 2, where we pass the number 2 instead of a string, fails with an AttributeError since we are using the split function which is a built-in method of the str data type. A manual way to fix this would be to check if str_input is an instance of str.
def do_something(str_input):
if isinstance(str_input, str):
split = str_input.split()
return " ".join([s.capitalize() for s in split])
Now this is cumbersome and inflexible at the same time. The Python developers came to the same conclusion and, starting with version 3.5, Python supports type hints. PEP 484 defines how to give the Python interpreter a hint about what type your function is expecting. The above function would, with type hints, look like this:
def do_something(str_input: str) -> str:
split = str_input.split()
return " ".join([s.capitalize() for s in split])
Now, while this is a great hint to you as a developer, PEP 484 explicitly states in its non-goals that type checking at runtime is left to third-party packages so even with the above type hints, a number can still be passed without causing a runtime failure until calling the split function on it.
One such third-party package is pydantic. With this little tool you can write Python data classes whose variables will be checked to match a specified type. To see this in action here is a (truncated) Device class I wrote for a little project this week:
from pydantic import BaseModel
from typing import List, Optional
class SipUrl(BaseModel):
pass
class Device(BaseModel):
id: str
displayName: str = None
sipUrls: List[SipUrl] = []
d = Device(id="test_id")
In the Device class, the id is specified as a member variable of type string and, since no default value is provided, is not optional. Trying to instantiate a Device object without specifying an id will result in a ValidationError that specifies that the id field is required but missing. The other two variables, displayName and sipUrls, have default values and work with custom classes as well as containers like lists.
Pydantic also adds some convenience functions like a .dict() and .JSON() function to turn your data model into a dictionary or JSON. The inverse functions are also present. You can use the parse_obj, parse_raw and parse_file functions to create an object from a dict, JSON string or file handler.
One last cool feature of pydantic is that it will convert variables when possible. Let’s say you specify a field to hold ints and pass ‘2’ which is a string. This should fail since ‘2’ is not an int, but the value can be parsed into an int automatically which is exactly what pydantic is going to do.
class TestModel(BaseModel):
id: int
t1 = TestModel(id=1)
t2 = TestModel(id='2')
print(type(t2.dict()['id']))
This code will work and checking the type of the id field will yield an integer.