My latest contribution to the world is Git Lint, a plug-in for git that allows you to pre-configure your linters and syntax checkers, and then run them all at once on only the things you've changed. About half of us still live in the command line, and I like being able to set-and-forget tools that make me a better developer.
Here are a few things I've learned along the way about Python projects.
1. Use A Project Template
Project templates provide a means to magically produce a lot of the boilerplate you're going to be producing anyway. I'm fond of Cookiecutter. While Git Lint started life as a single Bash script (later, a Hy script, and now a Python module), at some point I needed much more than just that: I needed documentation and testing. Up-to-date templates provide you with up-to-date tools: Tox, Travis, Sphinx, PyTest, Flake8, Twine, and a Makefile come pre-packaged with Cookiecutter's base template, and that's more than enough to launch most projects.
2. Setup.py is a beast
Getting setup.py to conform to my needs was a serious pain in the neck. It still doesn't work correctly when installing to Mac OSX because the Python libraries and the manual (man pages) are in two different locations. If I'm building a command line tool, I always try to provide man pages. It's usually the first place I look.
The manual pages also didn't show up reliably in the build process; I had to force it by adding it explicitly to the manifest, even though it included the docs tree by default.
Setup.py and man pages are NOT friends.
Getting the build to include man pages, which I require for any command-line utility, was truly a pain in the neck, and now every upload to pypi has a manual step where I figure out if I have a man page to deliver or not. It's truly painful.
3. Sphinx is a pain in the neck — but it's worth it for Github Pages
Sphinx, the documentation tool for Python, uses RST (reStructured Text), which has just about the worst imaginable syntax for external links I've ever wrestled with. Inconsistencies about mixing links and styles drove me out of my mind.
On the other hand, I now have what I consider to be a solid idiom for generating Github Pages (gh-pages) from Sphinx documentation. A branch named "gh-pages" that contains your documentation will automatically be converted into a documentation tree on Github Pages (github.io), and you can see the results for Git Lint. This tree looks completely different from your development trees, so don't get them confused, merge, or rebase them!
Simple generation of gh-pages
If you check out the Makefile, you'll see the idiom clearly: it checks out a complete copy of itself into a sub directory, builds the documentation, copies it back to the parent directory, and then fixes all the links (because Github Pages really doesn't like underscores in file and directory names). There's irony in that it uses Perl to do the fixing-- it's just what I knew, it was fast, and I always have both Python and Perl installed.
This, by the way, points to another issue: always use the same virtual environment wherever you work. My Macbook and my Linux box had different versions of Sphinx on them, and the resulting generated pages were different on both boxes, making git report "everything's changed!" when I went to fix a single typo in a link somewhere (I told you I hated those links).
It might be worth it to bag sphinx as a docker feature, or ensure that the version is locked down in your virtual environment.
4. Tox is amazing
Working with tox allowed me to be reassured that my code ran correctly every time, the first time. It did not catch other critical issues with installation, like the man page issue mentioned above, and that was painful to manage, but it did everything else.
5. Git Porcelain Zero is ridiculous
If you're not familiar with
git --porcelain, it's an argument that many of the status-oriented git commands have that changes the output to a stable, machine-readable form meant to be consumed by other tools. Git Lint uses it a lot.
git --porcelain command doesn't have any other guarantees: it doesn't guarantee filename sanity, or unicode compatibility. For that, there's
git --porcelain -z, which produces a report in which everything is null-terminated so weird filenames can be consumed. This would be fine if the output were columnar, but it's not always. The most egregious example I found was
git status --porcelain -z, which is usually three columns, but if there's an 'R' in the first column, then it's four columns-- 'R' means the operation is 'rename,' and the fourth column is the original name.
-z argument makes both the cell and the line terminators null, you have to parse positionally. And if you're parsing positionally and the number of positions can change, well, that's context-sensitive parsing. And it's ridiculous to have to put a context-sensitive parser into a small project like this. There was only one exceptional case here, so it's a small issue, but inconsistencies like this really bother me.
6. Git lint is amazing
All in all, this was one of those projects where I learned a lot about everything: git, python, unit testing, documentation, github, jekyll, reStructuredText, Cookiecutter, PyPi. All this knowledge poured into one small project.