Tuesday, June 20, 2017

JIRA for project finish time estimation

Software projects are notorious for their schedule overruns. One of the reasons is that project finish dates are usually based on wishful thinking.

Herewith I propose a data driven approach using JIRA: The most important thing is to have enough issues in place to make statistical analysis meaningful, let's say at least 30 issues. After that, we need to make sure that we close issues faster than we create them. Only after we satisfy these two conditions can we talk about reasonable finish time estimations.

Below is my proposed project status page that updates whenever issues change, i.e. you do not need to ask people about when the project will finish or what the completion percentage is, it is all done instantly and automatically. The manager will always have up to date estimations. Only after your team has sufficient practice with this system should you worry about velocity/burndown etc. charts.

Calculation of parameters:

Enough issues = Total nb of issues > 30 ? Green : Red

Issues decreasing = (nb of closed issues in last 3 months) > (nb of created issues in last 3 months) ? Green : Red

Avg. nb of issues closed per month = (nb of issues closed) / (nb of months passed from project start)

Months required to finish project = (nb of open issues)/(avg. nb of issues closed per month)

Estimated finish date = Todays date + Months required to finish project

Estimated completion = (nb of closed issues) / (total nb of issues) *100

Wednesday, April 05, 2017

Modular design

The most important software design principle is separating the system into independent modules. By modules, I mean separately compilable units. Benefits of modular software design:
  • It will be harder to introduce unintentional side effects like
    • Adding dependencies to code not related to the module.
    • Delete code that is not related to the module.
  • Modules can be designed separately which decreases mental load.
  • Each module will have its own unit tests, when unit tests are run, not the whole project but only that module will be built.
  • Each module will have its own string resource file instead of a giant resource file for the whole project.
  • Bad design (usually due to inexperience, time pressure or plain laziness) will be limited to the module in question and won’t be a project wide cancer.
  • There will be less merge problems.
  • Portions that have higher security requirements can be dealt with locally in modules instead of affecting the whole code.
  • Being able to test modules independently helps in identifying bugs like memory leaks.
  • It will be easier to define and assign tasks to developers. As long as the inter-module interfaces are clearly defined, integration will be easy.
  • It will be easier to estimate time and resource requirements.
  • If will be easier to reuse the module in other projects.
  • A module might be used standalone to do some tasks that does not require the whole system to be online.
  • If MVC was used, desktop GUI can be quickly converted to web front end and logic can be used as back-end and the whole application can be converted to a web application.

Friday, March 31, 2017

Android client connecting to java server example

As a case study on how to use a VPS, I bought a VPS from Amazon LightSail, wrote a server in Java and a client in Android:

  1. I selected the preconfigured Ubuntu option. It has no GUI, you have to do everything on the command line.
  2. I created a static IP. 
  3. In order to access it freely (ping etc.), go to networking tab, under firewall add “All TCP+UDP”
  4. On Ubuntu, I installed OpenJDK8:
    1. sudo apt-get update
    2. sudo apt-get install openjdk-8*
  5. I installed Putty to use as an SSH terminal
  6. I installed Filezilla for file transfers.
Server software:
  1. Wrote server in Java on my laptop and transferred the code to VPS via FileZilla.
  2. I used gson for object-JSON conversions. The latest gson jar file can be downloaded from here.
  3. I wrote a java build script in Ubuntu using nano.
    1. Note that if you write the script in Windows and transfer it to Ubuntu, it might not work because line endings in Windows and Linux are different.
    2. To make the file executable: chmod +x filename
  4. I built and run the server.
  5. To make sure that server keeps running when I disconnect putty, I used nohup and disown.
Android client software:
  1. Wrote client in Android Studio.
  2. I used gson for object-JSON conversions.
  3. I had to use AsyncTask for connection, otherwise I get NetworkOnMainThreadException
The source code for java server and android client is on GitHub.

  • Server: Handle multiple simultaneous connection requests

Saturday, March 18, 2017

Dealing with legacy Matlab code

A friend of mine has a project that works in Matlab Simulink 2007. He recently opened that simulink model in Matlab 2010 and could run it from Simulink without problems. But when he tried to transfer that model to xpc target for real time integration, the simulation hung and became unresponsive.

He and his team tried to find the problem by disabling Simulink blocks one at a time and in the end found the block that was causing problems. When they compared the 2007 version with the 2010 version, they could not find any differences which means that there is probably a backward compatibility issue that affects real time deployment. They had to manually recreate the whole model (more than hundred blocks) manually in Matlab 2010. Doing manual editing in code always carries the risk of breaking something. You might not know it unless you do tests with high coverage.

My first advice was to create a unit test framework so that you are sure that the new version yields the same outputs for the same inputs. A good strategy is to write scripts that vary inputs using unform distribution, run simulation on xpc target for every input, record outputs (or timeout when simulation hangs) and then compare outputs with previous results and create a report that highlights differences.

My second advice was to automate the update-to-newer-Matlab process by writing a program that automatically creates a new Simulink mdl file from the old one, beause it is easy to parse the mdl file which has an xml-look. Steps:
  1. Takes as input the depth of parsing. 1 = Copy the whole model. 2=Go one step deep and create separate mdl files for blocks in that depth 3=Go 2 steps deep etc.
  2. Parses the old Simulink mdl file and extracts the blocks
  3. Creates an mdl file in the new version format.
  4. Inserts the blocks into new mdl file(s)
  5. Runs unit tests on xpc target to see if the two versions have the same outputs (i.e. they are within 1e-15) neighborhood.
With this program, my friend can first check if for input depth =1, the model passes tests. If not, he can increase the depth until he can isolate the problem. Increasing depth on error could be automated too.

Saturday, March 11, 2017

Intelligence tests

I don't have much faith in intelligence tests. My definition of intelligence is successful adaption to new circumstances. Adaption and persistence determine your material success. How can you hope to test adaption, especially the social component in an hour long test?

Recently a friend of mine told me that his child's test score indicated his child being super intelligent but my friend didn't value the result much, although it was better than having an imbecile score. Our conclusion is that such tests mainly measure how much care the child was given by the parents. If parents have done a good job of parenting, I suspect any kid who does not have a mental condition will score highly in those tests.

Unrealistic proposal requests

Besides software development, my responsibilities also include assessing and answering simulation related proposal requests from other firms. Since simulation is usually poorly understood, the requester puts in sentences like "the simulation shall be realistic" without knowing how vague and overly ambitious that is.

Without going into philosphical details about what is "realistic", I can say that it involves a lot of real world testing and modeling. The budget that is available is usually less than 1% needed to fulfill realism. In that case we have two choices:
  1. Prepare a proposal that takes every possible risk into account and demand a billion dollars (same as saying NO).
  2. Think about what can be done with the given budget, write down your assumptions and limitations and present that as a proposal. This is the way to go if you want to avoid suffering.
The major cost factor in a simulation project is integration and testing. Your first question to any proposal request should be "how will you know that our simulation works as you expect". The answer will determine not only how much testing is needed but also how detailed your models should be. It is like unit tests driving design. If you don't get a satisfactory answer, write down in your proposal how and where you will test it.