Sunday, July 16, 2017

CI for C# projects: AppVeyor

I tried Travis CI for continuous integration of a C# WPF project and it failed because Travis CI uses Mono and that is not sufficent for WPF apps. So I tried AppVeyor and was successful. My minimalist appveyor.yml file is as follows:
image: Visual Studio 2017
configuration: Debug
before_build:
  - nuget restore

Friday, July 07, 2017

Difference between novice and experienced engineers

It has been said that good/experienced software engineers can be several orders of magnitude (i.e. 10x-100x) more productive than novice engineers. I think the main productivity improvement is the fact that experienced engineers are much better at detecting/avoiding bullshit:
It’s far more dangerous to assume people know what they’re talking about, than it is to assume they don’t and let them prove you wrong.
When I was a fresh engineer, I naturally did the work given to me by other people, without questioning if it was worth doing. As I got more experienced, I realized that most of the time people don't have thought through the job they are assigning to others. This results in getting stuck in dead ends, doing things that have no importance which result in lost time and motivation.

Nowadays, I am in my 20th year as an engineer. I pride myself in my analysis skills, ability to see signal through the noise and avoiding waste of resources. A lot of times I finish a job by demonstrating that the job in question is worthless/wrong. If you compare my lines of code per unit time, a novice might produce more. But that usually means producing garbage at a faster pace, creating more problems than solutions. That is why experienced people are worth their weight in gold.

Saturday, July 01, 2017

Using Travis CI for Java Projects

When using Travis CI for automated testing / continuous integration of Java projects created with the NetBeans IDE you have to do the following:
  • On NetBeans, Project/Properties/Sources make Source/Binary Format equal to JDK7. If you use JDK8, Travis CI ant will fail.
  • You must have at least one JUnit test.
  • You have to copy Junit and Hamcrest jars to project directory and add them under Libraries in NetBeans. If you leave them under Test Libraries, Travis CI won't be able to find them.
  • Commit changes to GitHub.
  • Add .travis.yml file to your github project root with the line language: java
  • Go to Travis CI and sign in with your GitHub account. 
  • Click on your Travis CI profile picture / Accounts. Your open source GitHub projects will be listed.
  • Enable your project for CI by clicking the button to its left.
  • Click on the project to go to build page which kicks off build.
  • If you have done everything correctly, you should see a success message at the end of the build page.
  • To add the build banner on top of build page, click on the banner, select Markdown and copy the link. 
  • On your GitHub page, edit README.md, paste the link and commit. 
You can see the build results for my snake project here.

Tuesday, June 20, 2017

JIRA for project finish time estimation

Software projects are notorious for their schedule overruns. One of the reasons is that project finish dates are usually based on wishful thinking.

Herewith I propose a data driven approach using JIRA: The most important thing is to have enough issues in place to make statistical analysis meaningful, let's say at least 30 issues. After that, we need to make sure that we close issues faster than we create them. Only after we satisfy these two conditions can we talk about reasonable finish time estimations.

Below is my proposed project status page that updates whenever issues change, i.e. you do not need to ask people about when the project will finish or what the completion percentage is, it is all done instantly and automatically. The manager will always have up to date estimations. Only after your team has sufficient practice with this system should you worry about velocity/burndown etc. charts.


Calculation of parameters:

Enough issues = Total nb of issues > 30 ? Green : Red

Issues decreasing = (nb of closed issues in last 3 months) > (nb of created issues in last 3 months) ? Green : Red

Avg. nb of issues closed per month = (nb of issues closed) / (nb of months passed from project start)

Months required to finish project = (nb of open issues)/(avg. nb of issues closed per month)

Estimated finish date = Todays date + Months required to finish project

Estimated completion = (nb of closed issues) / (total nb of issues) *100

Wednesday, April 05, 2017

Modular design

The most important software design principle is separating the system into independent modules. By modules, I mean separately compilable units. Benefits of modular software design:
  • It will be harder to introduce unintentional side effects like
    • Adding dependencies to code not related to the module.
    • Delete code that is not related to the module.
  • Modules can be designed separately which decreases mental load.
  • Each module will have its own unit tests, when unit tests are run, not the whole project but only that module will be built.
  • Each module will have its own string resource file instead of a giant resource file for the whole project.
  • Bad design (usually due to inexperience, time pressure or plain laziness) will be limited to the module in question and won’t be a project wide cancer.
  • There will be less merge problems.
  • Portions that have higher security requirements can be dealt with locally in modules instead of affecting the whole code.
  • Being able to test modules independently helps in identifying bugs like memory leaks.
  • It will be easier to define and assign tasks to developers. As long as the inter-module interfaces are clearly defined, integration will be easy.
  • It will be easier to estimate time and resource requirements.
  • If will be easier to reuse the module in other projects.
  • A module might be used standalone to do some tasks that does not require the whole system to be online.
  • If MVC was used, desktop GUI can be quickly converted to web front end and logic can be used as back-end and the whole application can be converted to a web application.

Friday, March 31, 2017

Android client connecting to java server example

As a case study on how to use a VPS, I bought a VPS from Amazon LightSail, wrote a server in Java and a client in Android:

VPS:
  1. I selected the preconfigured Ubuntu option. It has no GUI, you have to do everything on the command line.
  2. I created a static IP. 
  3. In order to access it freely (ping etc.), go to networking tab, under firewall add “All TCP+UDP”
  4. On Ubuntu, I installed OpenJDK8:
    1. sudo apt-get update
    2. sudo apt-get install openjdk-8*
  5. I installed Putty to use as an SSH terminal
  6. I installed Filezilla for file transfers.
Server software:
  1. Wrote server in Java on my laptop and transferred the code to VPS via FileZilla.
  2. I used gson for object-JSON conversions. The latest gson jar file can be downloaded from here.
  3. I wrote a java build script in Ubuntu using nano.
    1. Note that if you write the script in Windows and transfer it to Ubuntu, it might not work because line endings in Windows and Linux are different.
    2. To make the file executable: chmod +x filename
  4. I built and run the server.
  5. To make sure that server keeps running when I disconnect putty, I used nohup and disown.
Android client software:
  1. Wrote client in Android Studio.
  2. I used gson for object-JSON conversions.
  3. I had to use AsyncTask for connection, otherwise I get NetworkOnMainThreadException
The source code for java server and android client is on GitHub.

TODO:
  • Server: Handle multiple simultaneous connection requests

Saturday, March 18, 2017

Dealing with legacy Matlab code

A friend of mine has a project that works in Matlab Simulink 2007. He recently opened that simulink model in Matlab 2010 and could run it from Simulink without problems. But when he tried to transfer that model to xpc target for real time integration, the simulation hung and became unresponsive.

He and his team tried to find the problem by disabling Simulink blocks one at a time and in the end found the block that was causing problems. When they compared the 2007 version with the 2010 version, they could not find any differences which means that there is probably a backward compatibility issue that affects real time deployment. They had to manually recreate the whole model (more than hundred blocks) manually in Matlab 2010. Doing manual editing in code always carries the risk of breaking something. You might not know it unless you do tests with high coverage.

My first advice was to create a unit test framework so that you are sure that the new version yields the same outputs for the same inputs. A good strategy is to write scripts that vary inputs using unform distribution, run simulation on xpc target for every input, record outputs (or timeout when simulation hangs) and then compare outputs with previous results and create a report that highlights differences.

My second advice was to automate the update-to-newer-Matlab process by writing a program that automatically creates a new Simulink mdl file from the old one, beause it is easy to parse the mdl file which has an xml-look. Steps:
  1. Takes as input the depth of parsing. 1 = Copy the whole model. 2=Go one step deep and create separate mdl files for blocks in that depth 3=Go 2 steps deep etc.
  2. Parses the old Simulink mdl file and extracts the blocks
  3. Creates an mdl file in the new version format.
  4. Inserts the blocks into new mdl file(s)
  5. Runs unit tests on xpc target to see if the two versions have the same outputs (i.e. they are within 1e-15) neighborhood.
With this program, my friend can first check if for input depth =1, the model passes tests. If not, he can increase the depth until he can isolate the problem. Increasing depth on error could be automated too.

Saturday, March 11, 2017

Intelligence tests

I don't have much faith in intelligence tests. My definition of intelligence is successful adaption to new circumstances. Adaption and persistence determine your material success. How can you hope to test adaption, especially the social component in an hour long test?

Recently a friend of mine told me that his child's test score indicated his child being super intelligent but my friend didn't value the result much, although it was better than having an imbecile score. Our conclusion is that such tests mainly measure how much care the child was given by the parents. If parents have done a good job of parenting, I suspect any kid who does not have a mental condition will score highly in those tests.

Unrealistic proposal requests

Besides software development, my responsibilities also include assessing and answering simulation related proposal requests from other firms. Since simulation is usually poorly understood, the requester puts in sentences like "the simulation shall be realistic" without knowing how vague and overly ambitious that is.

Without going into philosphical details about what is "realistic", I can say that it involves a lot of real world testing and modeling. The budget that is available is usually less than 1% needed to fulfill realism. In that case we have two choices:
  1. Prepare a proposal that takes every possible risk into account and demand a billion dollars (same as saying NO).
  2. Think about what can be done with the given budget, write down your assumptions and limitations and present that as a proposal. This is the way to go if you want to avoid suffering.
The major cost factor in a simulation project is integration and testing. Your first question to any proposal request should be "how will you know that our simulation works as you expect". The answer will determine not only how much testing is needed but also how detailed your models should be. It is like unit tests driving design. If you don't get a satisfactory answer, write down in your proposal how and where you will test it.

Sunday, March 05, 2017

Software manager reading list

A few days ago, a manager friend of mine asked me for advice about what to do in a project that is near crisis, i.e. it has missed a deadline and looks like will be over budget.

Crises are common in software projects because their complexity usually is beyond our time and resource prediction capabilities. I offered my friend the following books and recommended reading them first before demanding that his team starts working overtime:
These books will ground him in reality and make him realize what kind of a mess he is in. I also told him that he should only tell his superiors the reality if he is certain that they are interested in the truth. A lot of times people want to hear only good news and are content with ignorance until it hits.

If you sense that your superiors want to live in fantasy land, never tell them the bad news because you will be labeled as a pessimist. Even after your predictions prove to be true in the end, you being labeled as a nuisance won't be washed away.

These books will help you understand what is going to happen and why. You might not be able to change the outside world, but you will have less stress inside, not feeling any guilt, knowing that you did the best you could.

Monday, February 27, 2017

Probability: Difference between "exactly..." and "at least..."

In probability, I have to pay close attention to the problem statement because I easily make errors. As an example, let's say I toss a fair coin n times. What is the probability of
  1. Exacly 1 head
  2. At least 1 head
The figure below has n as the x axis and the two different probabilites as y axis. As you can see, the "exactly 1 head" probability decreases with n while the "at least 1 head" probability increases with n.

Matlab script for generating the plot:
n = 1:16;
pExactly1 = n./2.^n;
plot(n, pExactly1); grid on
hold on
pAtLeast1 = 1-(1-0.5).^n;
plot(n, pAtLeast1)
legend('exactly 1 heads', 'at least 1 heads')
xlabel('n'); ylabel('p')
Note that you can also get probability of at least 1 heads with this sum: pExactly1Head + pExactly2Heads + ... + pExactlyNHeads, where
pExactlyKHeads = nchoosek(n,k) * ph^k * (1-ph)^(n-k)
Even simpler: 1 - pExactly0Heads = 1 - nchoosek(n,0) * ph^0 * (1-ph)^(n-0).

For more information, visit Khan Academy.

Car IoT

As aircraft have become software with wings, so are cars on their way to become software on wheels. Self driving and IoT have accelerated this trend. Yesterday, when I was talking with my friends we envisioned a design that could improve driving comfort: Cars could collect vibration data measured by their sensors and transmit it together with location information to servers. This data would indicate road quality. The server could process and broadcast data to all cars. Other cars could adjust suspension settings in advance that would maximize comfort on that road.

With this technique, only the first few cars that collected data would suffer and the rest would benefit. Data would be current at all times. I think Google Maps uses a similar approach to guess traffic congestion by collecting data from smartphones.

To collect data we need:
  • Vibration sensors on car.
  • Access to internet and access to central data server
To adapt to road conditions we need:
  • Adaptable suspension on car (i.e. actuators and sensors)
  • Access to internet and data server by car.
If we could protect privacy, we could automatically tap into the intelligence of the swarm.

Note to self: Do a simpler version of this concept as a hobby project.

How to evaluate personnel

It would be easy to evaluate performance if everybody (including the evaluator) was of high quality and there was no Dunning–Kruger effect. Your only criteria would be years of experience. Unfortunately, real life is more complex than that and there are large differences between employees.

The next best thing might be to have three bins: low, average, high. The low bin might be for fresh graduates who just started to work. The high bin consists of people who are highly productive and experienced.

You divide your staff according to their general level into these bins. Each bin has a lower and upper salary bound. Then you rank the people in each bin according to their productivity. I know that realistically measuring productivity is a tough problem but there is no escaping from it. For example, you have to decide if the person with 3 months of overtime had really productive overtimes or was just producing 8 hours of work in 12 hours. In each bin, at the top of the pyramid should be people with high productive overtime and at the bottom should be persons with no or low productive overtime. That way, you assure that everybody gets their fair share and productive individuals get their extra.

Tuesday, February 21, 2017

Android development tips

When developing Android apps with Android Studio, I get strange errors from time to time, whose solutions were not obvious to me:

Problem: When you create a Google Maps demo and run it using an emulator of API level 24 (e.g. Nexus 5 API 24), the app opens on the emulator with the message "...won’t run unless you update Google Play services".

Solution:  In \app\build.gradle file, com.google.android.gms:play-services version might be 10.2.0. Change it to 9.8.0 and your app will run fine.

Problem: In Android Studio, when I try to generate signed APK, I get "android execution failed for task validatesigningrelease" error.

Solution: Make sure you specify full path of key file.

Problem: When I try to generate signed APK, I get “Execution failed… method ID not in [0, 0xffff]: 65536” error.

Solution: If you are using Google Maps, limit the number of libraries you use by replacing in your gradle file the line compile ‘com.google.android.gms:play-services:9.8.0’ with
compile 'com.google.android.gms:play-services-maps:9.8.0' and
compile 'com.google.android.gms:play-services-location:9.8.0'

Sunday, February 19, 2017

Better WhatsApp Group Chat

Group chat is widely used in WhatsApp. As the size of the group gets bigger, communication is dominated by few people who usually generate a lot of noise. I had to leave groups because I was fed up with the endless nonesense of these few people. Arguing with them makes matters worse. Almost everyone I know has similar experiences.

I can silence a whole group in WhatsApp so that I do not get notifications. What I need is the ability to mark people in a group as important and unimportant. Messages from people that I mark important should be easy to spot when I scroll through the messages. They could have yellow background and bold font. Messages from people I mark as unimportant should be shown collapsed and become only visible in full when I click on the message. That would allow me to quickly see important messages.


Friday, January 20, 2017

My first Android app: Rescue Map

A friend of mine is a member of the search and rescue organization AKUT. He asked if there is an app that could draw the estimated victim region on a map according to victim category.

The category specifies at what speed the victim is expected to be moving. The app has to have offline map capability so that it can be used when there is no reliable internet connection.

I did a quick search on Google Play and could not find anything that did the job. So, I decided to write it myself as a first project in Android. The source code is on GitHub. I don't plan on publishing it to Google Play until my friend says that the app is really useful and others could benefit from it.

Racism test

Compared to older times, open racism is in decline. It takes on more light forms like "I love my culture/people". There is nothing wrong in loving certain things or having preferences. The critical point is how you view others: Do you just want to be left alone with your pure clan or do you think you also have a "natural" right to decide on others destinies because you are part of the master race/gender?

Likeable vs logical

In both professional and social interactions, being likeable is much more important than being right. I am amazed how much this fact is ignored by otherwise intelligent people. If people don't like you enough, they won't even listen to you which means that you are wasting your time constructing arguments. People will be meaner to you. Your name won't come up when there are opportunities or positions to fill. In short, your quality of life will suffer.

How can you be more likable (in order of difficulty):
  1. Smile and greet a lot.
  2. Don't talk if you have nothing nice or productive to say.
  3. Avoid ad hominem attacks and generalizations. Example: "You always do this because your are an emotionless prick!"
  4. Find mutually beneficial solutions to problems, even if it means to settle on a less-than-optimal solution.
  5. If you have to argue, keep calm and argue with unambiguous data. This implies that you have to collect data before the argument.
  6. Be sympathetic to others. People rarely mean exactly what they say. There are different dynamics at work that makes them behave as they do. If you take their words at face value, you might think that they are dumb or crazy. They are not. Their feelings might be hurt by something else, they might be tired etc. 
People have flaws, don't treat them like computers. The next time you are about to label others stupid because they are not convinced by your perfect arguments, ponder if they like you or not.

Book: Dark Sun: The Making Of The Hydrogen Bomb

Dark Sun: The Making Of The Hydrogen Bomb, Rhodes, Richard, documents the history and politics of hydrogen bomb development both in the USA and the Soviet Union. For the story of atomic bomb development see my previous post.

Book: Command and Control

Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety Schlosser, Eric, details how fragile nuclear weapons safety is.

Book: The Dictator's Handbook

The Dictator's Handbook: Why Bad Behavior is Almost Always Good Politics, de Mesquita, Bruce Bueno, Alastair Smith, explains why people in power might do seemingly irrational things and why loyalty is much more important than competence.

Similar previous blogpost: link 1, link 2

Related video 1:


Related video 2:


Related video 3:

Book: How to Fail at Almost Everything and Still Win Big

How to Fail at Almost Everything and Still Win Big by Scott Adams, is about trying different things, failing often, learning as much as possible from each failure and be ready when luck finds you. Highly recommended...