21 technologies transforming software development

 

A long time ago, developers wrote assembly code that ran fast and light. On good days, they had enough money in their budget to hire someone to toggle all those switches on the front of the machine to input their code. On bad days, they flipped the switches themselves. Life was simple: The software loaded data from memory, did some arithmetic, and sent it back. That was all.

Today, developers must work with teams spread across multiple continents where people speak different languages with different character sets and – this is the bad part – use different versions of the compiler. Some of the code is new, and some may be from decade-old libraries that may or may not come with source code. Building team spirit and slogging through the mess is only the beginning of what it means to be a programmer today.

The work involved in telling computers what to do is markedly different than it was even five years ago, and it’s quite possible that any Rip Van Winkle-like developer who slept through the past 10 years would be unable to function in the today’s computing world. Everything seems to be changing faster than ever.

Here are 20 technologies transforming the very nature of programming. They’re changing how we work with fellow developers, how we interact with our customers, and how we code. Don’t get caught asleep at the console.

Continuous integration

When you checked in code to a repository, there used to be enough time to catch your breath, have a cup of coffee, and maybe even go out to lunch. No more – code repositories are now tightly linked to continuous build systems that recompile your code, scrutinize your architecture, initiate hundreds of tests, and start flagging every potential error in your work. You won’t get five feet from your desk before your phone starts pinging you with new emails or text messages from the continuous build mechanism telling you what needs to be fixed. Back to work, slave, the continuous build machine has new tasks for you.

Smarter languages

The earliest computer languages were designed to make it easy to do anything with a computer. The latest want to make it hard or even impossible to do anything other than just the right thing. Over the years, the programming community has learned how people make mistakes and programs go bad. They’ve created a long list of bad practices and now some of the language designers are putting up guardrails and handing out straightjackets to keep us from following the last generation down the path to sins like null pointer dereferences or race conditions.

Rust, for instance, has one class of variables that don’t change. Yes, a variable that isn’t variable seems odd, but it’s one way to stop race conditions and make the code faster. There are hundreds of innovations like these and they’re allowing programmers to build grander piles of code for changing the world.

These constraints aren’t perfect, of course, and they can be annoying to talented programmers who instinctively avoid the bad techniques. But the newer languages are finding teams that enjoy the discipline and added structure.

Better databases

The first databases were miracles that saved the world’s programmers years of effort by offering a standard way of sticking information in big tables. Today’s databases shoulder those loads and many others like maintaining social networks, tracking locations, storing images, or more. They’ll do all of this while spreading the load over clusters of machines that may even live on different continents.

As interest explodes, the programmers have created dozens of databases that satisfy different niches. Do you want super-fast answers but you don’t care if tiny inconsistencies creep into the data? Then you want an in-memory database that doesn’t use transactions. Or is the data so valuable that it can’t be lost? Another class of database will automatically replicate the information across data centers in different time zones or even hemispheres. There are dozens of different options and sometimes you can find that one product satisfies different needs by flipping a few bits in some configuration file.

Frameworks

Standing on the shoulders of giants by reusing the work of others may not be a new idea, but it seems like it’s never been as dominant as it is today. Very little programming begins from scratch these days. The favored—and some might argue, best—approach is to grab the right framework, research the API, and start writing glue code to link together the parts of the API that make the most sense for the job. web pages aren’t built out of HTML or CSS anymore; the coding begins with Ext JS, ExpressJS, or some other collection of code that serves as a foundation.

Sure, you could be pioneering and build everything from scratch, but that would be suicide. There’s no way to catch up with all the work done by others. You’re not a craftsman—you’re a framework-tweaker. If you’re thinking of writing code yourself, stop and look for a framework that does it already.

Libraries

A close cousin to the framework is the library, a collection of routines so ubiquitous that coders can no longer live without it. Is it possible to write code for the browser without using jQuery? Does anyone even remember there’s a built-in function called GetElementByID? Nah, libraries like jQuery now rule every level of the stack.

People talk about their favorite languages, but that conversation says little about how they program. If you’re looking to hire someone, you need to ask about library knowledge. The game programmer may use C++, but the real question is whether the coder knows Allegro, Unity, Corona, or any of a number of other options. Knowledge of the library is as important as knowing the ins and outs of the language itself.

Node.js and JavaScript

Before some of you were born, web servers spit out static HTML. Then someone figured out how to create dynamic servers that could interact with databases. Every team needed one person to program the database in SQL, one person to write the server code in PHP or Java, and one person to design the HTML templates. Once everyone fell in love with AJAX and JavaScript running on the client, the sites needed yet another person to speak that language.

Now it’s all done in JavaScript. The browser, of course, still speaks JavaScript, but so do the server layer (Node.js) and the database layer (MongoDB and Couchbase). Even the HTML is often specified with JavaScript code for a framework like Angular or React that generates the HTML at the client.

Transpilers

Programmers are human too. If one wants to use language A, you can bet that the other one will want to use only B, or maybe C or D but never A. Deciding which one to use will take more time than doing the work. The beauty of many modern programming languages is that they can often be automatically rewritten in another programming language. The “transpilers” or “cross-compilers” suck out the basic structure of a program written in one language and turn it into something that runs on another. They’re not always perfect. They don’t always pick up every clever trick or every idiom, but sometimes they’re actually better than the original because they recognize an opportunity for optimization.

One of the biggest targets is JavaScript, the lingua franca for all browsers. Almost every language, from C++ to Python, can be fed into a converter that turns it into something that runs alongside the images and text in any webpage. With CoffeeScript and TypeScript, even JavaScript programmers have a way to turn improved JavaScript into regular JavaScript.

Code reviews and style rules

In the old days, a large program produced by a team of N programmers usually had at least N visibly distinct styles in the code and often a few more thanks to schizophrenic programmers. Those differences are fading as teams develop mechanisms for enforcing uniform styles so that the code is easier to understand because the idioms and design patterns are consistent.

Not every programmer is happy with the push for conformity. The most talented or expressive often feel like they’re forced to throttle their artistic powers. Sometimes the code reviews become vehicles for passive and occasionally openly aggressive confrontations that leave emotional scars and fractured teams. At their worst, they produce the committee-grade work one expects when too many cooks are stirring the pot.

Despite all of these problems, the managers continue to embrace the ideas because, well, it’s the only way to stop the bad programmers and weed out the faulty code. If it ends up thwarting the creativity of a few geniuses, well, that’s something we’ll just have to live with.

Preprocessors and linting

We old timers used to feel lucky if a compiler could spot an unused variable. Today, there’s an explosion of tools for preprocessing code to look for errors in logic or style. Did you fail to declare a variable? Or worse, did you use the wrong combination of spaces and tabs? The folks at Airbnb have become Internet famous by promulgating a style guide for JavaScript code. The set of rules even specify that a good programmer will put a space on both sides of a plus sign; anything else is bad. Is this harsh conformity good? The people who are writing the rules may be chuckling with glee like power-mad scientists. But at least they’re also giving us tools that will catch our dumb mistakes before we show them to the world.

Virtual machines

The days of writing code for real chunks of silicon are largely gone. Much of the code written today runs on virtual machines that translate your instructions into something understood by the chip. The Java Virtual Machine, the C#/.Net Virtual Machine, and now JavaScript engines end up being the main target for code.

The popularity of the VM is growing to absorb everything in the stack. In the past, if you wanted to create a new language, you would need to build the entire stack from pre-processor to register allocator. These days, new languages sit on top of the old virtual machines. Clojure, Scala, Jython, JRuby—they’re all piggybacking off Sun’s (now part of Oracle) great work in building the VM.

This same behavior is appearing in the browser world. Sure, you could create your own browser and language, or you could cross-compile it to be emulated in JavaScript. That’s what the folks did when they built cleaned-up tools like CoffeeScript. If this isn’t confusing enough, Google produced GWT (Google Web Toolkit) to convert Java to JavaScript.

APIs

Not so many years ago, programmers worried about data structures. They would pack all their information into blocks of bytes, count the bytes one by one, then make sure the values were placed the right distance from the pointer. Now, thank goodness, the compiler does most of that for us.

These days we work through a much more rigorous interface with a fancier name: an API. This is often on a completely different machine and may be run by a completely different company that is charging us for every call. Do you want a street address and a ZIP code turned into latitude and longitude? There’s an API for that, and it costs a few slivers of a penny to find each answer.

In most cases, the data doesn’t need to be so tightly packed. The old game of counting bytes has been replaced by parse-able data structures such as JSON or XML. You need to make sure you have the right punctuation in the right spot, but luckily there’s a library to handle that for you.

New user interfaces

The words “user interface” used to mean a decision between a command line and a GUI filled with pictures and icons for clicking. Now the televisions are “smart” enough to pull up websites, and all of the phones have a personal assistant like Siri or Cortana. Soon every kitchen and living room will have its own microphones waiting for someone to say “Alexa.” Can telepathy be far behind?

All of this is a challenge to programmers who must learn new libraries and tools that use different structures and metaphors. The area is new and that means that there are few established practices for programmers to borrow or build upon. They must create everything from scratch. The new paths can simplify things for the programmers. If the user wants to know the temperature, there’s no reason to create a full app or a pretty user interface. The voice can just speak the digits out loud and be done with it. The simplicity can be surprising to anyone who has spent an afternoon downloading the many gigabytes of Apple’s Xcode just to write a small app.

Browsers

There was once a time when people wrote software for desktops, software for servers, and software for devices, and it would all be different. Each had its own way of communicating with the user. Now everything goes through the browser. When I set up a local file server on my house to hold music, I go to a URL and work with a website. Widgets for Apple’s desktop have been written in JavaScript and HTML for years. Many cross-platform mobile apps begin as HTML and JavaScript that’s bundled with Apache Cordova.

Sure, there are holdouts. The best games are still custom work that doesn’t need a browser, but that’s changing, as more and more JavaScript developers figure out how to make magic in a browser window.

Docker and containers

Building a server used to be hard work. The programmers would get their code running, then send a memo to the team of server curators who’d install the right software. Sometimes they got the right libraries and sometimes they didn’t, but eventually we converged on something that worked.

Now application containers like Docker allow us to push a button and ship off a container with all the right libraries. If it runs on our test machine, it will almost certainly run on the server. Everything is bundled together, and most of the incompatibilities between our desktops and the server are gone.

Devops tools

Once upon a time, we installed software on a server—singular. Now we rent servers en masse, requiring dozens, hundreds, or even thousands of machines, many of which need to be provisioned on demand, full of fresh software—a job that can no longer be done effectively by hand.

Enter the “devops” trend and underlying tools such as Chef or Puppet or Kubernetes or Mesos are designed to maintain these servers for you. Push new software to the cloud and these tools handle the job of keeping all the computers running the same code. They automate what we used to do by hand for one machine.

Some services such as Google App Engine already handle this internally. All you need to do is give it your app, and the provisioning is automatic. You don’t even know what’s going on in the background; you merely get a bill for the amount of CPU cycles consumed.

Infrastructure as a service

Did I mention the teams of server curators? Those guys were fun to hang out with at lunch or after work, but now they’ve been abstracted away into the cloud layer, working as they do in a data center across the globe for another company that fancies itself a leader in the world of cloud this or cloud that. Few programmers need to ask the infrastructure team to build them a new server for a new project. They simply log into a website, push a button, and get a machine running for them. It’s so much easier, but these IaaS administration web pages won’t buy you a drink after work. Of course, that saves you from ever having to get the next round.

Platform as a service

Who builds their own website anymore? Instead, create an account on someone else’s website and customize it. All it takes is a few fields in a web form, and voilà, your new website does everything you wanted. It’s like uploading a cat video to YouTube or bidding on a Pez dispenser on eBay.

Of course, this is a bit of an overstatement. Many of the PaaS options today require a programmer’s sophistication to know what to put in each web form. Microsoft’s Azure, for instance, wants you to put in a few JavaScript functions that characterize how the website should respond. Then Azure wraps them up with the right libraries and starts them running on Node.js.

Secondary marketplaces

If you’re building a game, you could hire your own artists to create a stunning set of models. You might even hire a few programmers to add visual effects to make the game look cool. Or you could go shopping at secondary marketplaces like the Unity Asset Store and buy up all the pieces you need. As I write this, there’s a 33 percent markdown on the Tile A Dungeon Sewer Kit, “designed as a modular kit to build from small to large sewer game scenes.” The sale will probably be over by you time you read this and the price will be back up to $45. Who needs developers or artists with prices so low?

There are more and more effective marketplaces for plug-ins, extensions, libraries, and other add-ons. As with libraries and frameworks, here one doesn’t program so much as go shopping for the right pieces.

Social media portals

In the early days of the Internet, you would build your own website, cross your fingers, and hope people would find it. When they did, they simply had to remember your cool URL.

Alas, more and more of the web is being absorbed into big silos like Facebook and Salesforce. If you build your own website, you might turn it on and hear the sound of crickets because all of humanity is clicking away in Facebook or Salesforce.

The solution, of course, is to build a Facebook or Salesforce app. They’ll let you in and let you integrate with their platform to a point. But in the end, your app is an extra that could be limited or tossed aside with a wave of a hand. What choice do you have? You’re either a lackey to the big portals or you’re listening to crickets.

GitHub and social code sharing

Code-sharing sites may be the greatest contribution to the open source world. Before services like SourceForge came along, software was something you built on your own and shared on your own. If someone wanted a copy of the code, they came to you and you sent them a tar-ball if you felt like it.

Now code sharing is a social network. Sites like SourceForge and GitHub post all the code for everyone to see and update. They merge the process of maintaining, sharing, and commenting on the code in one easy-to-access place. You can read the code and suggest changes, all through one interface. Is it any wonder that many projects see tens or even hundreds of thousands of downloads each week? That would never be possible with the old model.

This model is now so dominant that most proprietary projects follow it. Sites like GitHub and BitBucket support themselves by selling nonpublic repositories that offer all the power of sharing, but within a limited permission group.

Performance monitoring

In the beginning, tracking the power of your code was simple. You printed out the time when the code began, then printed out the time when it ended. If you wanted to be fancy, you added a few extra calculations to do the subtraction for you.

That can’t cut it any longer. Many of the problems don’t occur on one machine. Adding a profiler to your code won’t reveal the real bottleneck, which could be caused by some weird interconnection or a sluggish database. Modern tools track the network calls for the network of software as well as the performance of individual modules. This is the only way to understand what is going right and going wrong.

This is just one important way of how the model of programming is morphing from a single machine to a network of connected tools that may or may not play well together.