Mikael Lundin Weblog

Configure React with different Environment Settings

Wed, 03 Nov 2021 20:39:02 +0000

I ran into this problem today, and after solving it was asked to blog about my findings.

I have a React application that is built to a Docker image and then statically hosted. I needed to configure the application with different OpenID Connect settings for test, staging and production environments. I couldn't just send environment variables to the Docker container, because the frontend application cannot read them.

I found a couple of different options.

Using .env files

Create React App has built-in support for .env files. When you build the application, the configuration options are bundled into the app with the app bundle. The problem is that dotenv doesn't work like you expect it to.

Basically it supports using different configurations for development npm start, production npm run build and testing npm test. It doesn't support environments like test, staging and production. It is also strongly discouraged to commit .env files to source control.

Documentation for .env files.
https://create-react-app.dev/docs/adding-custom-environment-variables/

Dynamic Rendering

Another technique I’ve used before, is to dynamically render the settings in a <script> tag in index.html.

<!DOCTYPE html>
<html lang="en">
  <head>
    <script>
      window.SERVER_DATA = <?php= json_encode($settings) ?>;
    </script>
  </head>
</html>

In your React application you can fetch the settings object from window.SERVER_DATA. By composing the settings object from environment variables, this lets you configure the React application from Docker container env vars.

The bad news is that your application is now dependent on external state, and that breaks portability. I don't like it.

I couldn't use the dynamic technique this time as my React files are statically hosted.

Some official documentation.
https://create-react-app.dev/docs/title-and-meta-tags#generating-dynamic-meta-tags-on-the-server

Fetch Configuration on Startup

I also have the option to fetch the configuration when application mounts.

function App() {
  const [settings, setSettings] = useState({});

  useEffect(() => {
    const fetchSettingsAsync = () => {
      fetch("/api/settings")
        .then((response) => response.json())
        .then((data) => setSettings(data));
    };

    fetchSettingsAsync();
  });

  // rest of the program
}

If you're hosting an API for your application already on localhost, this might be a cleaner solution, but I don't like that my app starts with showing a loading spinner. Not for data that could've been bundled with the application.

My Solution

I like the .env file approach. It is simple and integrates well into local development. However it doesn't support multiple environments out of the box. We can create a work-around by replacing the .env file in our CI.

First I create a .env.development file with reasonable defaults for local development. This can be committed to source control for simplified development setup.

Next, I create two files that I don't commit to source control.

.env-staging
.env-production

These files contain configurations for staging and production environment. As I've been using Azure DevOps I upload these files to the secure file area.

Next, I modify the build pipeline. Since Docker images are immutable I'll have to build one image for each environment.

In the build pipeline I download the secure file and add it as a secret to the docker build command. If using Docker Compose, secrets will not work during build step. An option here is to copy the file into the build context instead.

- task: DownloadSecureFile@1
  name: env
  inputs:
  secureFile: "env-$(envName)"

- bash: DOCKER_BUILDKIT=1 docker build --secret id=env,src=$(env.secureFilePath)  -t incaps/vaccinated:1.0.0-$(envName) .
  displayName: "Docker build"

In the Dockerfile, I copy the /run/secrets/env file into my source repository where the .env file normally would be. This file has preceedence to any .env variant like .env.development or .env.production.

# syntax = docker/dockerfile:1.1-experimental
# ...

RUN --mount=type=secret,id=env cp /run/secrets/env /src/.env
RUN npm run build

The app will now bundle with the environment specific configuration.

Downsides

Since I bundle the configuration into the React bundle, I need to build one docker image for each of my environments. This somewhat contradicts the purpose of Docker, build once - run everywhere, but it is the option that I favoured for my specific case.

Thoughts on Github Copilot

Tue, 02 Nov 2021 07:34:02 +0000

Github Copilot is a new service from Github that is currently in preview, and I’ve had the opportunity to take it for a spin. This service integrates with your IDE, helping you to write code by giving suggestions. It does this with the use of AI, which gives context sensitive, and very accurate results.

Go here and read about it to learn more.

My Experience

I was writing some elaborate text transformation logic and decided to try out Copilot. The quality of the suggestions were good enough that I didn’t have to write much code at all. I wrote some function signatures and could tab myself through the development. Functionality I expected would take several evenings of coding, was done in 30 minutes. The productivity boost was enormous.

In short what happened during this time

My workflow changed from writing code to directing Copilot to write the correct code for me
The generated code took advantage of the code I'd already written
There was very little need for adjustments or fixing the code that was generated
The generated code was working as expected on the first run
I didn’t spend time unit testing. Couldn't really see the point for the generated code

The Implications

I’m thinking about what will happen if this way of programming becomes the norm.

Productivity will sky rocket. Every developer will produce much more code
Instead of writing code, workflow will focus around generating the code that you want
There is no need to understand the code you generate as long as it works
Systems will become less maintainable, focus will shift to rewrite since it's cheaper
Manually writing code, will be considered as writing assembly without a compiler today

The big paradigm shift will be that system developers will drive the code generation to do what they want, instead of writing the code as we do today.

What do you think?

Cloud PC Development Environment - Windows365 Review

Fri, 06 Aug 2021 20:34:02 +0000

I've been using virtual machines for development since 2012. It started as an experiment, but I kept at it because I could easily see the benefits.

I could isolate each customer in a separate virtual machine
I could install customer specific software, like VPN clients, without screwing up my host
I could easily archive a customer development environment and resume it later
I could experiment with the development environment and easily roll back to a snapshot taken before the experiment

The downside of working in virtual machines are

They are slow, especially the Windows ones
They take up a lot of harddrive space, and they need to stay on the SSD for speed
They also make the host OS slow as they consume a lot of resources when they run

I’ve been looking a while for remote development environments suitable for development, and I did experiment with coding over RDP during 2013-2014. It worked okay, but it wasn't a game changer.

Running remote Linux environments has been mainstream for quite some time, but getting it to work well with X has been quirky at best. I still prefer to have my development Linux as a local virtual machine.

Windows365

In July Microsoft announced their Windows365, Cloud PC access on a subscription model like Office365. Their goal is to make it easy for enterprises to hand out a virtual machine sin the cloud to employees where you have the Office suite and Teams already installed.

I think it's a great idea, and it could really revolutionize remote working. But I’m not looking at it to write my e-mails. I want to use it for Visual Studio.

For this they have the following offering.

4 vCPU
16 GB RAM
128 GB Storage
$66.00 user/month

This is enough for running Visual Studio. If you want to, you can increase it with more cores, RAM and Storage, but it’ll soon get quite pricey. This was the machine configuration that I chose.

Review

I’ve been using this cloud machine as my primary development environment for the entire week and I’m positively surprised.

It was very easy to setup to get started. It took only a couple of minutes before the machine was created, and I could connect with a RDP profile that I easily downloaded for Microsoft Remote Desktop.

With Windows365 you can also connect with iPhone, iPad or just a browser. I tried the browser option just to get the feel of it, but I really want to use multiple screens, so Microsoft Remote Desktop is the client for me.

The Cloud PC is very fast. Installing Visual Studio was a breeze and quicker than ever. I rarely feel that the environment is slow. It is much more performant than my local virtual machines ever been.

I was expecting input lag from working over the Internet, and I did notice it on Monday when the service was new and everyone was trying it out at the same time. Since then it has become much better and I honestly forget that the computer I’m working on is not here in front of me, but in a data center somewhere else. It's actually that good.

I work with docker containers, and I had problems of running docker inside virtual machines before, however this was not a problem at all in my Cloud PC. Running Docker in Ubuntu WSL2 on Windows works like a charm. And the Docker Desktop for Windows also seems to be working as expected.

What strikes me is that I don’t need to carry around a machine anymore. If I'd go into an office, I could just connect to my development environment from any crappy office computer there. That would never happen since I love my MacBook Pro M1 too much, but the idea of it feels very liberating.

Downsides

It doesn’t replace all the functionality that I had with virtual machines. I can’t

Take snapshots and roll back
Archive an environment in order to return to it later

I actually have no idea how backups work, as I can’t see anything about it in the Windows365 control panel. What happens if I screw things up and want to roll back?

I would love the ability to just pause my Cloud PC and it would stop costing me money, but as it is now you need to remove the Windows365 license from your user which means that everything is gone and deleted.

There is a feature to reset your Cloud PC, but that would mean reinstalling all software and configure it from scratch. With local virtual machines I had the option of creating a template, that I’d use to quickly setup new machines. I could delete a development environment and setup a new one in 10 minutes.

These features are probably stuff that they will add later on.

My last complaint is about the price. I think the $66/month is a bit high and it means that I will remove my Cloud PC when I’m not using it. If you turn that coin, there is no large upfront cost like buying a physical PC, and the ability to cancel your subscription means that you can pick a higher end configuration and just cancel it those months you don’t need it, and still get a cheaper deal compared to buying a PC.

I will keep using my new Cloud PC. It really feels like the future to me. My MacBook Pro thanks me as well, as I haven't heard its fans for the whole week. :)

100 Interview Questions for Software Developers

Fri, 16 Jul 2021 20:55:02 +0000

This blog post is inspired by Jurgen Appelo 100 interview questions for Software Developers. I prefer to have a conversation instead of a knowledge test, so my questions do not have any correct answers. They should lead to conversation instead of validating knowledge.

Here we go.

Development Methods

How do you prefer to version control your code?
Do you follow any known patterns of version control?
What does continuous integration mean to you?
What is your definition of done?
How do you safely deliver code to production?
How do you make your production environment reproducible in test and dev?
How would you start troubleshooting an issue only visible in production?
Describe the steps involved for you to fix a bug
What is the importance of git commit messages and how do you format them?
How do you roll-back a failed deployment to production?
Production goes down because of a bad deployment. What would make you roll-back and what would make you roll-forward?
How do you manage assets that should not be committed to source control, like passwords or databases?
Where do you think code ownership should lie?
When should refactoring happen?
What code should be written by solitary programmers, and what code should be pair programmed or mob programmed?

Quality Assurance

What does quality mean for you?
How do you ensure quality in your delivery?
Who is testing your code?
What does testability mean to you?
What quality measures do you consider low hanging fruit?
Where do you draw the line between enough testing and too much testing?
Mention what statical code analysis tools you use
Do you write tests first, after or at the same time as the code?
What is your opinion on integration testing?
How do you deal with a test suite with brittle tests that require much maintenance?
How can you improve quality by communication?
How do you think that management can enable quality?
How do you measure quality?
What are your thoughts on technical debt?

Coding

What is your favorite language and why?
What do you consider a good framework?
What is your preferred way to document your code?
How do you make your code readable to others?
What do you consider maintainable code?
What are your most used development tools?
What is the worst feature of your favourite programming language?
What is the most common programming error people do?
How do you typically handle errors in your application?
Mention something you do to make your application secure.

DevOps

What’s your suggestion to improve deployment frequency?
What’s your suggestion to reduce time from you start working on a feature until its available to users?
What can we do to prepare for a system to be restored faster, when it goes down?
How would you propose we reduce number of releases that cause problems in production?
How do you verify the impact of your feature in production?
Where do you draw the line between development and operations?
How do you manage different configuration options for different environments?
How can you as a coder make your service easier to manage for operations?
What are the things that you believe should be automated?
What are the tools you need in order to deliver code to production once per day?
Do trunk-based development contribute to better software delivery?
How do we create predictability in our software delivery?
What’s required for you to feel inspired by your work?

Hosting

Which is your favourite cloud provider and why?
What does cloud native mean to you?
What is your relationship with Kubernetes?
What is your thoughts on serverless?
How do you measure uptime of your services?
How do you collect telemetry from your services?
Mention a way to improve availability of your services
How do you monitor the services you have released?
What would make you choose on-prem hosting over cloud?
What is a good way of managing SSL certificates?
What aspects do we need to consider regarding data protection laws?

Agile Software Development

What is your opinion on Scrum?
What is your opinion on user stories?
What is the most common impediment for you to be able to do your job?
How do you estimate correctly?
Which meetings do you consider necessary?
How do we make sure to maximise customer value?
What roles do you have in your optimal development team?
What activities do you think a team should do to improve its delivery?
The team will not be able to deliver in time. What is your proposed solution?
What do you expect from a project manager?
What does it mean to have a good team culture?
How do you follow up on progress in a project?

Software Design

What is your opinion on Microservices?
What is your opinion on SOLID?
What is your view on ORM frameworks?
What does object oriented programming mean to you?
In what capacity do you use functional programming?
Where do you draw the line between refactoring and rewrite?
Where do you draw the line between buying software or building it yourself?
What is your perspective on “best practices”?
How do you document your software design?
What tools do you use to design your software?

Other

What technological advancement do you consider a fad, and expect to be gone in 5 years?
How do you think we’ll be writing software in the future?
What will be the major challenges for our industry moving forward?
What do you think will happen with Machine Learning and AI?
What technology would you like to specialise in?
What technologies would you like to learn to broaden your expertise?
If you were not a software developer, what would you be doing instead?
What is, your opinion, the worst mistake done in history of computing?
What do you think is the most important traits/qualities for a developer to have?
How do you contribute to a good team culture?
What is required for you to feel satisfied with your work?
What can we do to prioritise diversity and promote an inclusive environment?
How do you learn new skills?
What does ethical coding mean to you, and how do you write code responsibly?
Does a software developer have to know how a computer works?

Please don’t ask your candidate all 100 questions :) and make sure that you ask non development related questions as well. Happy hiring!

It's about time

Thu, 27 May 2021 14:11:02 +0000

My last two years at Fitness24Seven I've been working in the role of Product Owner for our digital services. Never before have I worked with software projects where the product owner had software delivery experience. So I would like to share my thoughts on product ownership.

It's about time.

You are the Superhero!

You have a magic abilily to create time. Nobody else in the team can do this. A good project manager can save time, without making too many tradeoffs, but the product owner can create time out of thin air.

For a medium sized project of 3000 hours, there are lots of requirements and expectations. Your team will have to work 4 months to create the final product, but as you are aware lots of things may happen in 4 months. The only way to reduce the risk is to split the project into smaller parts.

As a product owner you split the project into two projects of each 2000 hours. You have now created 1000 hours without increasing the amount of work. What happens when you split work into smaller pieces is that you discover that you didn't know all the details before. You have discovered work that wasn't accounted for but still expected to be done within the 3000 hours.

A 2000 hour project is still a large project so you split it into two 1250 hour projects, and so it goes until you have found the smallest deliverable projects. You take all of your projects to the business people and ask them, "which one of these will provide most value to the business", and then you start working on that.

By reducing the size of the deliverable you have not only reduced the risk of the project, but also made sure that the value will be available to the business earlier, and you have created enough time for the team to make a good product. This is your superpower!

The smaller the better

Once you've found your appetite for dividing projects into smaller deployable things there is no end to it. You start splitting user stories into smaller deployable pieces in order to deliver sooner (not faster). The new functionality generates more value, as you get it to your users earlier.

When we start splitting user stories we do not split it into frontend and backend parts. We split the work into complete vertical slices of functionality. Instead of building an e-shop we will start by delivering a product catalogue. Next we deliver the ability to buy and pick up at a service point. Next we deliver shipping option with a shipping partner.

By developing and delivering in small chunks we also learn continuously what the users want and can change the prioritization from what we've learned. This improves the value output of your product as you do not have to guess what your customer wants. This would never have been possible with the 3000 hours of fixed scope.

After a while you'll find that projects are too big pieces to deliver at once. You start splitting it into even smaller parts to be deployed weekly or even daily. "Projects" turn into a bureaucracy for budget planning, a ceremonial to keep business people content, while your team deliver value in a continuous stream of functionality.

Scope it down

Your stakeholders will come with wishlists of functionality they want included in your product. The UX designer will create modern digital solutions that your users will love. The backend developers will design a system that will scale to 100 million users.

It is your job to scope it down.

Find the core functionality that will generate value. Remove all the extra finesse and tell your developers to make the easiest solution possible. You will deploy it within days and validate if it did generate expected value. You decide to add that extra UX cruft, deploy and find that users enjoy it. The feature starts to behave sluggish, so you ask your backend developers to make it work for 1000 users. They deploy and your users are happy.

What I have found is when you've delivered the core functionality, there's always something else the business wants you to focus on. Since you work in small chunks this is not a problem and you can change priorities in order to deliver as much value as possible.

Sometimes the core functionality doesn't feel like the feature is done. But it is good enough for now. If there is a need for it to improve you will get back to it later.

How to do this

There are a couple of things you need before you can start doing this.

You need to make deployments trivial
You need to get your development team in the boat with you
The business needs to trust you're doing the right thing
You need to approach product ownership as a full time job
You need to focus on quality and don't let the team take shortcuts

Reading list

The Lean Startup, Eric Ries
Accelerate, Building and Scaling High Performing Technology Organizations by Nicole Forsgren PhD, Jez Humble and Gene Kim
Get in the boat, Pat Bodin

Ten Principles of Software Projects

Thu, 20 May 2021 14:11:02 +0000

I've been working for Fitness24Seven for 2,5 years developing custom software solutions. We have been running a lot of projects, both short and long ones and come to value a set of principles that help us create great deliveries.

I was thinking I would share those with you.

1. We work in a continuous stream

When a task is done we release it. And then we bring in a new task. We still have iterations, but only for hosting demo and retrospective with regular intervals.

We do not

Work in what Scrum define as time boxed sprints
Commit to a backlog that should be done at the end of the sprint
Stress about imaginary deadlines at the end of the sprint
Cancel work because it wasn't done at the end of the sprint
Create a sprint backlog that is closed for modifications during the sprint

2. We optimize for the throughput, not for utilization

We try to have as little parallel work as possible. We carefully manage our work in progress to make sure that all current tasks are actively being worked on.

We do not

Bring in more work before we have completed something on the board
Cancel a task that is being worked on, instead we finish it
Bring in more work when a team member has spare time. We try to find ways for that team member to help out with the work on the board.

3. We do not keep a backlog in the traditional sense

The most important task to work on is constantly changing. When we finish one task, we will pick the next task that is most valued by our stakeholders.

We do not

Hoard user stories in a backlog, if they haven't been picked up in 6 months we delete them
Fret on old bugs. If nobody has fixed the bug after 6 months of reporting we delete them
Spend time grooming our backlog. The Kondo Principle applies - if you do not love it, delete it

4. We work without estimates

Estimates take time from doing the work, it takes time from the project manager to follow up, and it provides very little value. If the product owner needs to know if Task A is more complex than Task B, he can just ask the team.

We do not

Spend time in planning meetings
Follow up on how much planned work is left on a task
Measure or follow up on pace
Keep a burn down chart

5. Our Project Manager is a servant of the team

It is the purpose of the project manager to clear a path so the team can work without impediments. Problems should be solved before they affect the work on progress.

We do not

Follow up on who works on what, as long as tasks are being completed
Follow up on how many hours anyone has worked
Measure the team by key performance indicators
Keep people responsible for failures

6. We love planning, but do not follow a plan

In planning we do all the research we need to start a project, even estimates to figure out a reasonable budget. When the project starts we throw away that plan because it is already outdated. What we bring with us into the project is the learnings from making the plan.

We do not

Follow up a project based on the original scope
Bring estimates from budget planning into the project
Make promises to the business that we cannot keep
Let an outdated plan affect our decisions in the project

7. We build our team on trust, not control

We assume that everyone is here to do their best. That is why we do not ask what you did yesterday, but rather what has happened with this task since yesterday. What the team members did yesterday is irrelevant.

We do not

Measure how our team members perform
Follow up on how much time has been spent on a task
Require team members to be glued to their keyboard
Ask anyone to stay late to finish their work
Crunch, ever

8. We help each other out, even if it's not our job

Most important is that tasks are progressing, not that a frontend developer only do frontend code or a tester only do testing. A backend developer can create a solution architecture and a frontend developer can review the backend developer's code by just saying "explain it to me".

We do not

Claim ownership of our role or our tasks
Let tasks sit dormant waiting for an individual to be free
Communicate in silos
Let prestige stop us from doing what's best for the project

9. We design solutions just in time for implementation

There is a short due date to design. That is why we do our requirements gathering, graphical design, solution architecture first when it's time to start working on the task.

We do not

Start designing until we know that the task will be worked on
Do technical design or investigations before the work has started
Groom the backlog with requirement gathering

10. We always stay excellent towards each other

Personality traits are #1 priority when we bring in new employees or consultants to the team. If we find people that we like, the work will be much more fun and easy going. The technologies are always something you can learn on the job if you have the right attitude.

We do not

Require people to be 100% effective on day one
Hire people based on their technical prowess
Care about titles

The most important traits are care, compassion, humbleness and the ability to be excellent towards your team mates.

Development Methods

Mon, 07 Jan 2019 19:57:02 +0000

I'm a TDD practitioner, in the sense that I write my tests first. I am a test first developer, a red-green-refactor cultist. I'm very open about it and it offends people. They want to discuss tests with me, tell me that they've tried this unit test thing and it doesn't work, and they want me to convince them that unit testing is the only way so they can complain later about my dogmatism.

It's just like being a vegetarian.

But this was not intended to be a rant. Instead an observation that people seem to think I practice TDD because of testing, while I see unit tests as a byproduct of the way I work. The same people look horrified when I tell them to delete tests that doesn't work. The purpose of TDD is not to build a test suite. It is to provide a method of developing software.

Starting from a blank IDE. How do you start implementing a feature? What is your method to go from nothing to something?

In TDD we start with a test, or rather, we start with thinking. - What do I want this code to do? - Then I implement a test that verify it - Then I implement code to satisfy the test - Then I refactor - And then I start thinking again what I want this code to do.

It is a development method that provides very well thought programs. And it provides a test suite. But if you tell me that the test suite cost more to maintain than the value we get through regression, I will gladly delete it. But I will add a new test again for the next feature that is being developed.

This is the development method that I prefer. How do other people do it? How do they go from zero to something?

Most people I asked this question to doesn't know how they do it, they can't articulate it. But if you sit and observe what they do, you start to notice that they do have a method. Here are some methods that I've observed.

Design Up Front Development

I once were in a team that made every piece of code in UML before they started writing it. There was an architect that would draft out sequence diagrams and hand over to the developers. The developers would then take a look at the sequence diagram and base their implementation on that.

There was no requirement that the sequence diagram had to be implemented exactly as specified in the UML to code. Most often the coder would find things the sequence diagram was missing and fill in the blanks. The sequence diagram was never treated as a source of truth. After the implementation it was pretty much thrown away.

The UML was just a way to think about the problem before putting it on paper, in code. Big Design Up Front has been so criticised in the industry for a long time, but this way of communicating the problem and the solution worked very well.

Top Down Development

Developers that are very UI oriented tend to use expressive programming like HTML to implement the user interface first. After they have the inputs, the buttons, the navigation structure, they start working themselves down to the code that is going to provide the data. I call this top-down-development. You start from the UI and the interactions and you write code to satisfy it. Adobe Flash was the pinnacle of Top Down Development.

Solutions that are implemented top-down are always very user oriented, and they often run into trouble where they do not consider how the backing services expect data to be managed. You'll have beautifully implemented frontends that are slow, unmaintainable because of the lacking backends.

Bottom Up Development

You tell a bottom-up developer to integrate the e-commerce solution with a payment provider. He will study the e-commerce interface in detail and he will map that exactly onto the payment provider so it fits in a beautiful way. However he will never along the way ask how it's supposed to be used.

This is when you get UIs that looks like a Google Form on top of a Microsoft Access database. All the fields are there and you can almost sense the database table structure from the way that input fields are positioned.

Combining a top-down developer with a bottom-up developer is of course a very good idea. A lot of interesting things will happen when they try to interface to one another.

Code Fix Development

One of the most common way of developing software I see when I meet other developers is the code-fix development. They rarely realise they are practicing a development method but they do.

First they write up a draft of a program that they think will work. Then they run it. The program will most often crash, or not do what it was supposed to do, so they fix the problem and run it again. The program will fail on a different thing and they will fix it again. This will go on until they are satisfied with the result.

Depending on how much thinking is involved in this process, you will get different results. This is what managers mean when they say good developers and bad developers.

I think we can agree that this way of developing software by trial and error is not a very good one, but it is incredibly common. There will always be issues left with the code that reaches production. I get the sense that the code is never quite done. There is always one more fix needed.

Debug Driven Development

In .NET the debug tools has become so powerful that you can use them all the time, and it is very common for .NET developers to write code with the debugger. It works like this.

Write some code
Debug it and make sure it did what it was supposed to do
Rinse and repeat

This is a much more thorough development method than code-fix development, because you actually stop and analyse what your code does before you try to run it, but it is also a very slow process, and neither does it provide a step for thinking or refactoring your code.

The risk that the debug-driven developer run into is that they feel they need to refactor in the middle of debugging, so they start refactor the code before they've finished what they were doing. Feature changes and refactoring intermingles in an uncontrolled way, which is often obvious looking at the commit history.

I know many developers that write code this way and it works well for small solutions, but as soon as the code grows big it will have tons of technical debt because thinking is not a required step and neither is refactoring.

Summary

These are the development methods that I've observed in the wild. What development methods do you practice or observed?

Static Generated Websites are Shit

Thu, 06 Sep 2018 05:34:35 +0000

Maybe I missed a 'the' in the title, maybe I didn't.

My blog is old. It dates back to 2007 when I started it on Wordpress. One great thing about a blog platform like Wordpress is that even if the CMS platform will age and become harder and harder to update, the application will keep on working. You will still be able to update with content even if your PHP is 3 major versions behind.

I moved off Wordpress because the platform didn't feel secure. It kept nagging me about updates and it was hard to make changes the way I wanted to.

So I moved the whole thing over to Orchard CMS. That was a fun project because I was learning a new CMS and it was my first adventure on Azure. However it turned out that Azure was way too expensive for my small little blog, and Orchard CMS was a pain to keep updated.

My last move to Jekyll was great! Finally! No database, only markdown files. However similiar problems started to appear a year after the initial release. I had reinstalled my computer and didn't have the whole Jekyll setup. I found out the painful way that finding and installing old Jekyll dependencies was a real pain. It would take a whole day to setup an environment to start blogging, every time I would reinstall my system. So for long periods of time had no setup where I was able to update the blog.

It struck me that I need to script the setup. I don't want a blog virtual machine. I want to be able to write blog posts on both of my Macs, but I really don't want to install Ruby on any of them. So I concluded that I need to create a Docker container that have all the dependencies to generate my blog.

It did take a week, but here is my Dockerfile.

All this crap are things that I would need to figure out every time I'd setup a new environment for my blog. Now I can just run the following command

docker build -t miklund/tailcalloptimized:1.0 .

..and I have a complete environment for developing, writing and deploying. That is pretty neat.

Cloudfare

Another thing that has happened during my absense is that Google's now punishing you in the search result for hosting over HTTP.

I'm all for security, but I have not prioritized hosting my static content over HTTPS because there is no real threat here. The only plausable thing I can see happening is that a router infected with malware could inject links/ads into my content. That might concern me if my blog had thousands of visits per month, or representing a legitimate business, but not really for my rants about javascript.

Anyway, being punished in the search result for hosting over HTTP does concern me, but at the time of setting up my static blog the only option was jumping through a lot of hoops at Amazon which would require an investment that wasn't worth it by a long shot.

Today Cloudfare is offering a service for websites where they proxy your content and serve it over HTTPS. That is pretty great, and since I don't have any high requirements I get away with their free plan (but I would be willing to pay $5 for their service if they start charging).

Google Analytics

When I created the Jekyll blog I added Google Analytics. Why do you add GA to a site? Well, to get some insight on how many users you have.

In the passing years Google has become more and more evil. They collect all information they can about every person on the internet in order to target advertisement and sell the information to the highest bidder. It is services like Google Analytics that allow them follow and collect everything you do around the internet. I don't want to be a part of that problem, so I'm removing Google Analytics from my site.

I'd rather be blind on my site's statistics, than intruding you my visitors' privacy.

To protect yourself on the internet, remember to

Browse Twitter, Google, Facebook in incognito mode
Regularly clear browser history/cache/cookies
Use a VPN, always
Don't use Google Chrome
Search the internet with an engine that doesn't track you

Cheers!

Recursively Search and Update a JSON/JavaScript Object

Sun, 02 Sep 2018 12:15:32 +0000

Hello my blog. Sorry I haven't updated you in a while, but I've been busy. However it seems like I have a reason to start writing again, so here we go.

The other day I came across the following problem.

I have a JSON export of a data schema from inRiver, and I have a CSV file with translations for each property in inRiver. The only thing I need to do is to update the JSON export with the values from the CSV file.

Easy! Or not so easy. I quickly figured out that C# was not the language for this task. I would be better off using a dynamic language like javascript.

The gist here is not about reading or parsing CSV or JSON, but rather how to search and update a big javascript object. If it were XML, I would just have used XSLT and be done in a snap.

Let's look at the end result.

This code will recurse through an object graph, looking for an object with matching ID property. If it finds it, it will run the update function on that object.

The important thing to notice in this code is that javascript is always copy by value. We cannot just recurse down to the part we want to change and update it, because then we would update a copy of the object. Instead we need to make sure that we write back each recursion to the object graph to get the change in the end.

This is how we use it!

Happy Coding!

Creating a WebAssembly binary and running it in a browser

Sun, 19 Jun 2016 12:47:32 +0000

WebAssembly, is the promise of an intermediate language of the web, or JavaScript as an Assembly langauge if you want to. It really came from the AsmJS initiative, where a static subset of javascript was found to run much much faster than the full dynamic language. WebAssembly is the next iteration of that where a new assembler language has been designed to run on top of the javascript engine. This comes with a couple of promises

It will run faster than ordinary JavaScript
It will load faster than AsmJS because it doesn't have to be interpreted from text
It has a binary format that will be smaller than anything text based
It will, in the future, also be debuggable in the browser

The most uses we have seen of asmjs so far has been compiling C/C++ into asmjs in order to run these in the browser. We have all seen the Unity 3D demos and been impressed. For me, WebAssembly is the assembler language of the web, and this provides a unique opportunity to create new languages that will target the web and compile to a real IL language, instead of compiling to the less efficient javascript.

This blog post will take a look at the binary format of WebAssembly, how to read it, write it and run it.

WebAssembly text syntax

In order to better understand the WebAssembly (from now on called wasm) we're going to take a look at the sample module for this blog post.

(module
  (type $0 (func (param i32 i32) (result i32)))
  (export "add" $add)
  (func $add (type $0) (param $var$0 i32) (param $var$1 i32) (result i32)
    (i32.add
      (get_local $var$0)
      (get_local $var$1)
    )
  )
)

This looks a little bit like LISP syntax. It is the textual representation of the module in an s-expression syntax. The module exports one function called add, that will add two integer numbers together.

The type section contains any function definition that are used in our module. It will be referenced to later.
The export section contains any functions that should be exported, and able to be called from outside this module.
The code section starts with a function declaration. This is the declaration of our function "add". It will only accept types of i32 (32 bit integer) and will add them together by the operator i32.add.

WAsm only has 4 types.

i32 (32 bit integer)
i64 (64 bit integer)
f32 (32 bit float)
f64 (64 bit float)

Any other type will need to be constructed artificially by reading and writing raw memory, but this is out of scope for this blog post.

Reading the Wasm binary format

By going through the specification of the binary format I have written a wasm binary from scratch. You don't have to do this, there are excellent libraries like binaryen that can help you transform a S-expression syntax directly to wasm binary. However, I wanted to do this from javascript and in that the tools are scarse.

Here's our program.

00000000  00 61 73 6d 0b 00 00 00  04 74 79 70 65 87 80 80  |.asm.....type...|
00000010  80 00 01 40 02 01 01 01  01 08 66 75 6e 63 74 69  |...@......functi|
00000020  6f 6e 82 80 80 80 00 01  00 06 6d 65 6d 6f 72 79  |on........memory|
00000030  85 80 80 80 00 80 02 80  02 01 06 65 78 70 6f 72  |...........expor|
00000040  74 86 80 80 80 00 01 00  03 61 64 64 04 63 6f 64  |t........add.cod|
00000050  65 8c 80 80 80 00 01 86  80 80 80 00 00 14 00 14  |e...............|
00000060  01 40 04 6e 61 6d 65 86  80 80 80 00 01 03 61 64  |.@.name.......ad|
00000070  64 00                                             |d.|

It has a preamble and 6 sections as we can see in the clear text representation.

preamble
type
function
memory
export
code
name

Each section starts with the name of that section and a count on how many bytes long the section is in total.

Preamble

The preamble is the first 8 bytes of the file. The first 4 bytes is a uint32 'magic number' 0x6d736100 that spells out '\0asm' in ASCII. The point of this magic number is to quickly determine if this file is a wasm module or not.

The next four bytes is a uint32 number that represents the version of the wasm specification this file was constructed with. In this case it is version 11, or 0x0000000b in hex.

Type section

All sections are optional, so when they appear they need to be correctly labeled. We start with the type section.

The first byte 0x04 tells the intepreter that the name of this section is the next 4 bytes. So 0x74 0x79 0x70 0x65 are not surprisingly the ASCII codes for the word type. Following that is a varuint32 telling how many bytes this section is, and by some reason I haven't figured out, (maybe because I'm reading the specs for version 10 and this file format is version 11) this varuint32 in LEB128 format is padded to all 4 bytes and an extra byte for the sign 0x0080808087. The translation of this to pure English is that the type section is 7 bytes long.

The type section is defined as follows

0x01 There is one type defined in this section
0x40 This is a function type that is defined
0x02 The function type has two parameters
0x01 First parameter is of type i32
0x01 Second parameter is of type i32
0x01 The function returns a result
0x01 The result is of type i32

What we're doing here is registering the function signature of i32 add(arg1 : i32, arg2 : i32) in the type section of the wasm binary.

This section is now complete and the next section follows directly there after.

Function section

Again the starting 0x08 tells us the identifier of this section is 8 characters and the following bytes 0x66 0x75 0x6e 0x63 0x74 0x69 0x6f 0x6e spells out the word function. The following varuint32 value 0x0080808082 tells us that this section is two bytes long.

0x01 There is one function we want to aknowledge.
0x00 The function of interest has index 0 in the type section.

That's it. We point out the function signature from the type section.

Memory section

The memory section starts with 0x06 that specifies that this section's identifier is the next six characters 0x06 0x6d 0x65 0x6d 0x6f 0x72 0x79. Those spell out the name memory.

The following varuint32 value 0x0080808085 tells us that this section is 5 bytes long.

0x0280 Tells the intepreter that the initial size of memory allocated by this module should be 256 pages of 64KiB memory. That is 16 MB memory.
0x0280 Tells the intepreter that the maximun size of memory this module is allowed to allocate is 256 pages of 64KiB memory.
0x01 Specifies that this memory is exported and visible outside the module.

I'm not so sure about this section and I need to play around with it more to find out the implications of the different settings here. These settings were just something I came up with for default until I've figured it out.

Export section

The export section starts with 0x06 that specifies that the following identifier is six characters long 0x65 0x78 0x70 0x6f 0x72 0x74 spelling out export. The following varuint32 value 0x0080808086 tells us that this section is 6 bytes long.

0x01 Specifies how many exports there will be
0x00 Specifies the index of the first export in the function table
0x03 The identifier of the exported function is 3 bytes long
0x61 0x64 0x64 Is the name of the exported function in ASCII, meaning add

This section makes the add function visible outside this module by exporting it. It is through this exported interface that we will call the add function from javascript.

Code section

This is the fun section.

The code section starts with 0x04 that specifies that the identifier for this section is four characters 0x63 0x6f 0x64 0x65 and it spells out code. The following varuint32 value 0x008080808c tells us that this section is 12 bytes long.

It might be good to revisit that AST of our program now.

(module
  (type $0 (func (param i32 i32) (result i32)))
  (export "add" $add)
  (func $add (type $0) (param $var$0 i32) (param $var$1 i32) (result i32)
    (i32.add
      (get_local $var$0)
      (get_local $var$1)
    )
  )
)

Everything except the body of the $add function has already been written down. The code section will contain the body part. Each function call has its own HEX opcode, that you'll be aware of. But most importantly, this is binary coded post-order, so that first the left node is written, then right now, and last the parent node.

Let's take a look

0x01 There is one function body.
0x0080808086 This first function body is 6 bytes long
0x00 There are 0 local variables
0x14 The opcode for the left get_local call
0x00 Get the 0 index parameter to this function
0x14 The opcode for the right get_local call
0x01 Get the 1 index parameter to this function
0x40 The opcode for i32.add

As you can see the code is almost written backward. This is the effect of rendering the AST post-order.

Name section

The name section starts with 0x04 indicating that the identifier is 4 characters long, namely 0x6e 0x61 0x6d 0x65 which is the word name translated from ASCII.

The following varuint32 value 0x0080808086 specify that the length of this section is 6 bytes.

0x01 There is 1 name entry
0x03 The first name is 3 characters long
0x61 0x64 0x64 The name add
0x00 There are 0 local names defined

I'm not really sure what a local name is, but I guess I will find out. And with this we have written all 114 bytes that conclude our simple wasm module.

Writing a WebAssembly binary module

I did not write this WebAssembly by hand in an HEX editor. I wrote me some javascript to generate the code for me, but before I did that I took a look at what data types are used in this format.

uint8 - single-byte unsigned integer
uint32 - four-byte little endian unsigned integer
varint32 - singed LEB128 variable-length integer, limited to int32 values
varuint1 - unsigned LEB128 variable-length integer, limited to values 0 or 1
varuint7 - unsigned LEB128 variable-length integer, limited to values 0 to 127
varuint32 - unsigned LEB128 variable-length integer limited to uint32 values
varint64 - signed LEB128 variable length integer, limited to int64 values
value_type - single-byte unsigned integer encoded as [1, i32; 2, i64; 3, f32; 4, f64]

My first reaction to this was this ain't too bad and the second was what is an LEB128?. It is a special type of encoding that you can use for compressing unused binary space. If you have an int32 and store a value less than 256, you only need one byte of that integer. The three other bytes are excessive. LEB128 let's you encode that binary so you only have to store the bytes you're using of the data type.

With the unsignedLEB128 I had to append a padding, for the values that define length of a section. I hope I can remove this in the future.

With this in place, it was pretty easy to define the data types.

And with the data types in place it was straight forward creating the binary code. It is however heavily commented and it repeats itself, but this is because I did it as a learning excercise. I would extract some methods if I were to make it formal and use it as a library.

With this, all we need to do is to write the binary array to hdd.

Testing the WebAssembly binary in a browser

We have created a binary wasm file, but how do we test it? The information on this was pretty scarse until I found this demo page. You can currently run wasm files in experimental modes on following browsers

Chrome Canary, open chrome://flags/#enable-webassembly
Firefox Nightly, open about:config and set javascript.options.wasm to true
Download preview of Microsoft Edge

If you have a wasm enabled browser you can go here and use the following binary to try it out.

The source for loading and running the wasm module is pretty straight forward.