Cool Work Environment

Parts of this were shot on my floor of my building.

http://apple.co/business-capital-one

To find out more, visit https://developer.capitalone.com/community/?io=true

Writing Tools

Read this: http://thesweetsetup.com/apps/our-favorite-pro-writing-app-for-mac/

What I have been doing instead of using these sophisticated, integrated writing tools?

I use OmniOutliner. https://www.omnigroup.com/omnioutliner I've used it for years. It does a lot of things. Most notably, I can create multiple columns so that I can create page budgets for outlines. Acquisition Editors like this. Except, of course, they like it as an DOCX file, which requires a bit of manual juggling to produce.

I use BBEdit and KomodoEdit for a the bulk of my writing. http://www.barebones.com/products/bbedit/index.html
https://www.activestate.com/komodo-ide/downloads/edit

"But wait," you say, "those are text editors."

(Or, more dismissively, "there are merely text editors.")

Correct.  I use RST markup and write in Unicode text.  I use tools to convert the RST text to a variety of other binary formats. See http://docutils.sourceforge.net/docs/user/tools.html for a list of tools. This is what I often use:

• rst2odt creates ODT files which can be then converted to DOCX.
• rst2html creates web pages.
• rst2epub creates ePub files. From there, Kindlegen can convert to Kindle-unique format. https://www.amazon.com/gp/feature.html?docId=1000765211 

How is this better than a tool like Scrivener? It depends -- as always -- on what you're trying to optimize. My pipeline has the (dubious) advantage of being very inexpensive. Except for OmniOutliner and BBEdit, it's all community-edition, free software. If cheap is your goal, I've got cheap.

The cool part is this.

The Mac OS X desktop is an integrated writing environment. I have browser, outliner, writing tool, publishing tool, etc., etc., all readily and immediately available. The "look and feel" isn't consistent, but I'm not sure that's a show-stopper.

The biggest difficulty?

BBEdit doesn't enable the Mac OS X grammar checker. Really. It's switched off. The grammar checker is sometimes handy for preventing a large number of common, dumb writing mistakes. BBEdit shows the word count, which is very helpful for some kinds of writing. I wind up using a second app (i.e. the built-in Mac OS X TextEdit) to make a grammar check pass.

I think, however, the hacker-friendly free-and-open-source tool chain may have reached the end of its service life.

Why Not Use Word?

"After all," you say, "MS-Word does everything."

Agreed. It does everything badly and confusingly. (1) The outliner is hard to use and is firmly tied to the text in a way that breaks outlines all the time. (What's that paragraph doing there? Why is it the wrong outline level?) (2) There are too many useless features. The presence of "advanced" mode is a UX nightmare come true. (3) The character-mode and paragraph-mode formatting rules are baffling (and break the outlining.) (4) The styles are essentially invisible: you have to click on the text and check the style side-bar to be sure that the (invisible) markup is actually right.

The worst thing is that publishers have house style sheets for MS-Word that drive the publishing pipeline. This means that writing involves a weird step where I have to apply the publishers styles to things that are **very** clearly annotated with RST markup. You have to review each word. The words may look right, but have the wrong style applied. This is extremely tiresome to get right.

I intend to stick with plain-text markup. Scrivener supports MultiMarkdown. It's not RST, but it seems to be as rich with built-in semantic categories.

The "Build Script" Idea

In compiled languages, the build script or makefile is pretty important. Java has maven (and gradle and ant) for this job.

Python doesn't really have much for this. Mostly because it's needless.

However.

Some folks like the idea of a build script. I've been asked for suggestions.

First and foremost: Go Slow. A build script is not essential. It's barely even helpful. Python isn't Java. There's no maven/gradle/ant nonsense because it isn't necessary. Make is a poor choice of tools for reasons we'll see below.

For folks new to Python, here's the step that's sometimes important.

python setup.py sdist bdist_wheel upload

This uses the source distribution tools (sdist) to build a "wheel" out of the source code. That's the only thing that's important, and even that's optional. The source is all that really exists, and a Git Pull is the only thing that's truly required.

Really. There's no compilation, and there's no reason to do any processing prior to uploading source.

For folks experienced with Python, this may be obvious. For folks not so experienced, it's difficult to emphasize enough that Python is just source. No "class" files. No "jar" files. No "war" files. No "ear" files. None of that. A wheel is a Zip archive that follows some simple conventions.

Some Preliminary Steps

A modicum of care is a good idea before simply uploading something. There are a few steps that make some sense.

1. Run pylint to check for obvious code problems. A low pylint score indicates that the code needs to be cleaned up. There's no magically ideal number, but with a few judicious "disable" comments, it's easy to get to 10.00.
2. Run mypy to check the type hints. If mypy complains, you've got potentially serious problems.
3. Run py.test and get a coverage report. There's no magically perfect test coverage number: more is better. Even 100% line-of-code coverage doesn't necessarily mean that all of the potential combinations of logic paths have been covered.
4. Run sphinx to create documentation.

Only py.test has a simple pass-fail aspect. If the unit tests don't pass: that's a clear problem. 

The Script

Using make doesn't work out terribly well. It can be used, but it seems to me to be too confusing to set up properly.

Why? Because we don't have the kind of simple file relationships with which make works out so nicely. If we had simple *.c -> *.o -> *.ar kinds of relationships, make would be perfect. We don't have that, and this seems to make make more trouble than it's worth.  Both pylint and py.test keep history as well as produce reports. Sphinx is make-like already, which is why I'm leery of layering on the complexity.

My preference is something like this:

import pytest
from pylint import epylint as lint
import sphinx
from mypy.api import api

(pylint_stdout, pylint_stderr) = lint.py_run('*.py', return_std=True)
print(pylint_stdout.getvalue())

result = mypy.api.run('*.py')

pytest.main(["futurize_both/tests"])

sphinx.main(['source', 'build/html', '-b', 'singlehtml'])

The point here is to simply run the four tools and then look at the output to see what needs to be fixed. Circumstances will dictate changes to the parameters being used. New features will need different reports than bug fixes. Some parts of a project will have different focus than other parts. Conversion from Python 2 to Python 3 will indicate a shift in focus, also.

The idea of a one-size-fits-all script seems inappropriate. These tools are sophisticated. Each has a distinctive feature set. Tweaking the parameters by editing the build script seems like a simple, flexible solution. I'm not comfortable defining parameter-parsing options for this, since each project I work on seems to be unique.

Important. Right now, mypy-lang in the PyPI repository and mypy in GitHub differ. The GitHub version includes an api module; the PyPI release does not include this. This script may not work for you, depending on which mypy release you're using. This will change in the future, making things nicer. Until then, you may want to run mypy "the hard way" using subprocess.check_call().

In enterprise software development environments, it can make sense to set some thresholds for pylint and pytest coverage. It is very helpful to include type hints everywhere, also. In this context, it might make sense to parse the output from lint, mypy, and py.test to stop processing if some quality thresholds are met.

As noted above: Go Slow. This kind of tool automation isn't required and might actually be harmful if done badly. Arguing over pylint metrics isn't as helpful as writing unit test cases. I worry about teams developing an inappropriate focus on pylint or coverage reports -- and the associated numerology -- to the exclusion of sensible automated testing.

I think tools like https://pypi.python.org/pypi/pytest-bdd might be of more value than a simplistic "automated" tool chain. Automation doesn't seem as helpful as clarity in test design. I like the BDD idea with Gherkin test specifications because the Given-When-Then story outline seems to be very helpful for test design.

On Waiting to Write "Serious Code"

Someone told me they weren't yet ready to write "serious code." They needed to spend more time doing something that's not coding.

I'm unclear on what they were doing. It appears they have some barriers that I can't see.

They had sample data. They had a problem statement. They had an existing solution that was not very good. I couldn't see any reason for waiting. Indeed, I can't figure out what "serious" code is. Does that mean there's frivolous code?

Because there was a previous solution, they had a minimum viable product already defined: it has to do what the previous version did, only be better in some way. One could trivially transform the previous product into unit test cases and an acceptance test case. Few things could be more amenable to coding than having test cases.

Since everything necessary seemed to be in place, I had a complete brain cramp when they mentioned they weren't yet ready to write "serious" code. "Serious?" Seriously?

It appears that this developer suffers from a bad case of Fear of Code™. I know some common sources of this fear.

1. Waterfall Project Experience (WPE™.) Old people (like me,) who started in Waterfall World, were told that we had to produce mountains of design before we produced any code. No one knew why in any precise way. Indeed, there's ample evidence that too much design is simply a way to introduce noise into the process. In spite of real questions, some folks think that you can write a design so detailed that a coder can just type in the code from the design. (This level of design is isomorphic to code; to avoid ambiguity it must be written as code.)
2. Relational Database Hegemony (RDH™.) Folks (like me,) who were DBA's, know that databases require a lot of design and a lot of review before they can be created. Writing stored procedures requires even more design and review time. You don't just slap an SP out there. It might be "bad" or "create problems." Also, when you insist on DBA's writing application code, it takes super-detailed, code-level design details. In effect, you must write the code for the DBA to write your code back to you. 
3. One and Done (OAD™.) Some people like to feel that they can write code once and it can be a thing of beauty and a joy forever. The idea of a rewrite is anathema to these people. While this is obviously silly, people still like the conceit that they can produce some prototype code that will be a proper part of every future release forever and always. It's not possible to make all of the decisions the first time regarding adoption and scaling and user preferences. Your prototype code will get replaced eventually: get over it. Write the prototype, get funding, move forward. Don't dither trying to make a bunch of future-oriented decisions based on a future you cannot actually foresee. You can't "future-proof" your code.
4. Learnings are Expensive (LAE™.) You can find people that think that the sequence of (spike, POC, version 0, version 1) is too expensive. They are sure that learning is a project drag, since no "tangible" results are created by learning. This means that they don't value intellectual property or knowledge work, either; an attitude is actually destructive to the organization. Knowledge is everything: software captures knowledge: a spike followed by a POC followed by version zero will arrive on the scene more quickly than any alternative strategy. Don't waste time trying to write version 1 from a position of ignorance.
5. Tools are Expensive (TAE™.) Some people feel that -- since tools are expensive -- they should be used rarely. Back in the olden days, when a compiler took many minutes to produce an error report, you had to be sure the code was good. (I'm old enough that I remember when compiles took hours. Really.) Those days are gone. Most compilers today work at the "speed of light" -- if they were any faster, you couldn't tell, because you can't click any faster. For dynamic languages, like Python, the speed with which code can be emitted makes all tool considerations quaint and silly.
6. Diagram it to Death (DTD™.) Rather than write code, some folks would rather talk about writing code. To them, email, powerpoint, and whiteboard are cheaper than coding. This is a false economy. Nothing is saved by avoiding code. Time is wasted drawing diagrams of things at a level of detail that mirrors the code. Pictures aren't bad in general. Detailed pictures are simply a stalling tactic.

I find it frustrating when people search for excuses to avoid simply creating code. While I see a number of sources, there are many counter-arguments available. 

1. Waterfall is dead. Make something minimal that works for this sprint. Call it a "spike" if that makes you happier. Clean it up in the next sprint. Create value early. Expand on the features later.
2. Databases are free now. SQLite and similar products mean that we can prototype a database without waiting around for DBA's to give us permission to make progress. Build the database now, get something that works. Rework the database as your understanding of the problem matures. Rework the database as the problem itself matures and morphs. Nothing is static; the universe is expanding; do something now.
3. No code lasts forever. Waiting around to create some kind of perfect value one time only is perfect silliness. Create value early and often. Discarding code means you're making progress. If you think it's important, write "draft" on every electronic document which might get changed. (Hint: version numbers are smarter than putting "draft" everywhere.)
4. A spike and code happens more quickly than code. It's a matter of technical risk: unfinished work is an "exposure" -- an unrealized investment. Failing soon is better than researching extensively in an effort prevent a failure that could have been found quickly.
5. Use a dynamic language and avoid all overheads.
6. Keep the diagrams high-level. Code is the only way to meaningfully capture details. Code endures better than some out-of-date Visio file that's in Sharepoint completely disconnected from GitHub.

It's imperative to break down the roadblocks. All "pre-coding" activities are little more than emotional props: knock them down and start coding.

The Value of Microsoft's Tools

See Andrew Binstock's "Windows 8: Microsoft's Development Re-Do".

The costs of these migrations has been enormous and continues to accumulate...

I can only rub my hands with glee and engage in shameless "I Told You So" self-congratulations.

Only you can prevent being held hostage by Microsoft.

More than once, I've observed that a strategy of using only proprietary tools would be expensive and complex.  And every time, the folks I was talking to trivialized my concerns as hardly worth considering.

I've seen orphaned software: it only compiles on an old version of Visual Studio.   I've seen software orphaned so badly that it can only be compiled on one creaky old PC.  The cost to convert was so astronomical that the customer preferred to hope for a product to arise somewhere in the marketplace.  When no suitable product appeared over the decades, the problem reached palpable Pants On Fire (POF) levels of panic.  All due to the hidden costs of Microsoft's tools.

I've even been told that VB is a terrible language, but Visual Studio makes it acceptable.

Language, Tools, Chickens, Eggs, Java and Python

Too much of programming is intimately tied up with the tools to support the development of the software.

Example 1. I was told -- with absolute and fierce conviction -- that VB may suck as a language, but Visual Studio more than makes up for the obvious problems. For some people, Tools Trump Language. Sadly, I've also had customers with ancient code they could no longer compile or maintain because the tools were out of support.

On Stack Overflow, you can read questions like this: "What IDE to use for Python?". In spite of this question's immense popularity, it gets re-asked all the time. Search for "Python IDE" to see endless duplicates. One of the most common duplicate forms of this question asks (or demands) code completion. As if there are folks who cannot write code without code completion.

Chickens and Eggs

The issue with sophisticated IDE's (like Eclipse, NetBeans, and even Komodo) is that you have to learn the tools before learning the language. Until you know something about the language, the tools, of course, are useless. Worse, Eclipse is for "enterprise" applications and is so fat with bells (and whistles) that it's hard to determine what to use and what it means.

So the tool is a prerequisite for the language. But the language is a prerequisite for the tool.

How to cut the Gordian Knot?

First Principles

Irrespective of the "Visual Studio makes VB not suck" crowd, language comes first -- and last -- and fills all the spaces in between.

Language is everything. Software is merely encoded knowledge. The language of that encoding is how we determine meaning; how we argue about correctness, adaptability, maintainability and security. Tools don't endure -- they come and go -- but the language remains.

The only thing more important than the language is the data itself. But that's another rant.

Proof, of course, is available everyone except in VB circles. For non-proprietary languages (Java, Python, etc., etc.) there are a large number of competing tools. One language many tools. Take the hint. Language is important.

Yes, some tools are so flexible, they cover several languages. But there's no universal tool any more than there's a universal language. And the bias is clearly very, very many tools for a given language and only a few languages for a given tool.

How To Start

Language comes first.

For Python, that's easy. Run Python, type code at the >>> prompt, and you're learning. Python comes with IDLE which is a minimalist IDE. It will get anyone started. Later, they can try other IDE's.

For Java, however, that's not that easy. It isn't however, impossible to get started. It's just challenging.

Option 1 -- Bare Knuckles. It's possible to edit text and run the javac compiler to learn a great deal of Java without an IDE. It's not a bad idea. It will get complex to manage projects with more than a few files.

Eventually that's what Ant, Maven and SCons are for. But that's not a good place to start. Again, the tools don't make sense until you start writing things big enough that the tools actually help.

Option 2 -- Succession of IDE's. It's probably best to start with a very simple IDE for Java. Something like Komodo Edit, TextMate or BBEdit. There are a lot of choices, but the idea is to find something little more than a text editor with a few tools. I've used these and like their relative simplicity.

The JavaWIDE toolset might be helpful. I haven't used it, but some folks suggest that it simplifies the language learning. Later a "regular" desktop IDE can be used.

Later, one can move to NetBeans or Eclipse.

Classrooms and Autodidacts

In the classroom, it's easy to demonstrate NetBeans and answer questions.

For auto-didacts, however, choosing the wrong tool leads to endless confusion. The chicken and egg issue isn't clarified by wasting time trying to install and use a tool that's too sophisticated for a n00b.

N00b autodidacts really need to start with a simple text-editor. They need to use `javac` to compile and `java` to run the resulting class. For the first week or two, this will do. Once past the fundamentals, however, IDE selection can start to make sense. A BBEdit/TextMate/Komodo thing should be next. This is good for -- perhaps a year or more. Then, when doing "real" programming, a heavier-weight tool makes sense.