Enabling technologies for IoT development

Small scale / hobby IoT development has become much easier due to the following:

• It is possible to buy electronic components in small numbers cheaply and reliably from China. It is a miracle of logistics that I get a couple of transistors for a couple of bucks, shipped to my doorstep from the other side of the world in two weeks!
• Arduino, it's IDE and tons of libraries for all sorts of devices saves you from dealing with details of device drivers.
• Machine learning with Tensorflow, OpenCV, and Raspberry Pi.
• Cheap and reliable web services like AWS.
• Easy to use server environments like NodeJS.
• Last but not least, YouTube gets you up and running quickly.

We live in amazing times where everyone can tinker with electronics because the barriers to entry are so low now. Even my father who is 73 years old asked me the other day where he can get some breadboards! Imagine what the youngsters of today will come up with in the future.

Transistors

Differences between NPN & PNP Transistors: "...an NPN transistor turns on when the current flows through  the base of the transistor... A PNP transistor turns on when there is no current at the base of the transistor [i.e. the base is connected to the ground]." Usually, the middle pin (#2) is the base but there are transistors like the TIP120 whose base is pin 1, check the datasheet.

You can connect transistors in parallel to handle currents that a single transistor can't.

PWM of a signal

Pulse Width Modulation can be used to convert an analog signal to digital or mimic an analog signal with digital (e.g. Arduino's analogWrite()). PWM basically converts the signal sample values into pulse width values (amplitude of pulses is constant). Below is a Matlab script that demonstrates PWM of a sine wave.

Software bug target

Any complex software will have bugs in it, no matter how hard you try to fix them all. When testing software, aim for bug free operation in 95% of all operational cases. Note that I say "all operational cases" and not "all possible cases" because possbilities might be infinite due to combinatorial explosion and 95% of inifinity still infinity!

Users / domain experts can tell you realistic edge cases. If you don't constrain your bug target this way, you might spend a lot of time on fixing bugs/optimizing of unimportant scenarios. That is why having a freqent iteration development methodology (e.g. agile) and lots of tests with real users is very important in building a reliable product.

Mixing C++ and C and calling a C/C++ dll from Java

In Visual Studio, it is possible to mix C++ and C code, C++ files should have cpp extension, C files should have C extension, the compiler takes care of the rest. When including a C header file inside a C++ file, wrap it inside extern "C { #include "somfile.h"}. Otherwise, you will get LNK2019 (the dreaded "unresolved external symbol" error).

When you want to create a dll from this mixed project to be used with Java, in your wrapper header file of cpp implementation, you have to wrap the header body inside #ifdef __cplusplus extern "C" { #endif ...  #ifdef __cplusplus } #endif. Otherwise you will be able load the dll in Java but when you call a dll method, you will get UnsatisfiedLinkError.

When naming your functions in C/C++ Java interface functions section, do not use underscores at the end of the function name like "Java_package_name_myFunction_rad_s()" because in Java interface functions, undescores are only used as package name separators. If you ignore this, you will get UnsatisfiedLinkError and wonder why.

The cost of refactoring

Today a colleague was proposing a design change to the backend of a GUI heavy application with the main rationale that the backend would be more elegant and the change would be easy to implement. I told him that the real cost of refactoring is the necessity to repeat all the manual UI tests (our automated Macro Scheduler tests are not ready yet). A GUI application has many paths to failure and we have been developing this particular application for a couple of years now. Changing the backend would certainly break some edge cases and from the users perspective, the app would have regressed.

The only acceptable reason for a backend refactoring would be if we had lots of functionality to add in the short term and the change would make such additions a breeze, being a net saving in effort, in addition to being elegant.

Baby / child sleep training

Last weekend we met with friends who have a couple of months old baby. They asked us about sleep training. We told them that we first tried it according to the book but then changed it dramatically.

The main problem was that our son, who was 1 year old at the time of training start, resisted being laid down into the crib. Resisted means that he cried and wept with a terrified "I am going to die" look on his face and arms extended towards you begging for help for 45 minutes! We could not bring ourselves to leave him alone in his room like that. So, we laid besides him in the crib until he fell asleep (took about an hour) and sneaked away.

After a couple of months, we put some pillows beside his crib and waited there for him to fall asleep which again took about 1 hour each day. Every half an hour, I changed places with my spouse. It sometimes took 1.5 hours. After he fell asleep he usually woke up when we put him into the crib and the whole procedure lasted for another half an hour. Trying to carefully lift and place him into the crib was bad for our backs too.

Finally we decided to buy an adult size matress and laid it on the floor of his room and gave up on the crib altogether. At bed time, we cuddled him, played children's music, and when he fell asleep, we simply left the room without the crib hassle. Win-win :)

Why HTTPS?

Recently I converted my web site from HTTP to HTTPS. Reasons:

1. Chrome flags all HTTP sites as unsecure by default which is annoying.
2. To protect login information from man in the middle attacks.
3. For Progressive Web Apps (web apps that can run offline)
4. For accepting online payment.

The basic steps of converting your site to HTTPS:

1. Buy a domain name and redirect it to your server IP.
2. Install Nginx on your server and configure it to forward traffic to relevant ports.
3. Obtain a TLS/SSL certificate and install it.
4. Add a cron job to automatically renew the certificate.

Complexity of IoT with web UI

One of my on going projects collects data from IoT sensors and displays them on a web page. The things I had to learn were quite a lot. Here is a short list:

1. VPS, Ubuntu, Putty, Filezilla
2. Nodejs, Javascript, HTML, CSS, socket.io
3. Arduino IDE, libraries for sensor, board etc.
4. Sending data from sensor to VPS
5. Custom hardware design for rugged case, smaller form factor and minimal power consumption (done by a friend of mine).
6. Obtaining a domain name and redirecting it to VPS IP
7. Nginx as a reverse proxy
8. HTTPS, certification
9. Login, register
10. Online payment
11. Database

Best way to expand your network

The best way to expand your network is to work on something. Besides learning new things, you will also meet like minded people, exchange ideas and expand your network.

How bureaucracies get out of control

Recently I had to spend more than an hour to convince some otherwise very smart people that we should not have QA approval to close a bug. Having such an extra step caused developers to refrain from using the type "bug". Instead, they started to use the more generic "issue" type. The result was not being able to see which issues were bugs. For me, bugs are the more important issues to work on. When you label an issue as a bug, you get a nice red exclamation mark to the left of the issue on your board.

My personal experience is that it is quite easy to add something to an existing procedure but 10 times more difficult to remove something. This bias leads to bureaucracy getting larger, more complicated and less useful in time. This is the exact opposite of "perfection is attained not when there is nothing more to add, but when there is nothing more to remove" [Antoine de Saint Exupéry]

Simple login demo with NodeJS

I wrote a simple / bare bones login demo with NodeJS. It uses cookies to remember the user. Note that this is the absolute minimum and should not be used in applications requiring security.

Note that in login.html we have to use HTTP POST (not GET. This line in login.html: form action="/login" method="post" ) so that user name and pass is not displayed in browser address bar!

Modifying Toys

As a hobby, I like playing with Arduinos and having the ability to control real world devices like sensors and motors. Recently I decided to add a Cylon/KITT eye to my son's old robot toy:

First I bought an Arduino Pro Micro and an addressable LED strip. I cut the LED strip to fit into the robot's head, wrote a sketch to simulate Cylon eye and tested the setup:

Then I made sure that it worked with 3 AA batteries:

Then I squeezed in the LED strip into the robot's head and tested once more:

Then I verified that it worked with the robot's batteries:

Finally I assembled the whole robot:

The hardest part was fitting the pro micro and all the extra cables into the robot's body.

What's the point of all of this you ask? First of all, I like doing such things. And secondly... Watch Eli's video: The Value of "Hobbies" for Tech Professionals.

Binary search tree height

As part of HackerRank 30 Days of Code Challenge, I solved the binsary search tree height problem (day 22). My solution is not the most efficient/prettiest solution but it is mine (!):

Building a NetBeans java project without installing NetBeans

NetBeans projects that use the Layout class require swing-layout.jar, otherwise you will get the error message "package org.jdesktop.layout does not exist". To build a NetBeans project on a computer without installing NetBeans, you have to copy the swing-layout.jar and add the following to project.properties:
file.references.swing-layout.jar = <path of jar>
java.classpath =
    ${libs.swing-layout.classpath}:\
    ${file.reference.swing-layout.jar}

Also make sure that there are no trailing spaces, especially in the java.classpath section. And finally keep the build-impl.xml file.

Software time estimation

Software project time and effort estimation is difficult. One of the difficulties arise from writing low quality components. These components incur technical debt (or they might just be a mess), break the system at a later time when no one remembers their details, and require frequent revisits. Poor components are usually difficult to understand too. All this results in a lot of unbudgeted effort and delays. Even if you estimated the time for completing the messy version of the component, you would not be able to take into account the additional cost required to get to a clean/stable version.

The corollary is that you have to focus on quality before you spend any time on schedule estimation. To achieve acceptable quality, write unit tests, do code reviews (and at least rudimentary algorithm documentation) and use static analysis tools. Only after you have unit tests, documentation, code reviews and static analysis in place, you might try to guess when the project finishes.

Child education

Today we were talking with friends about our favourite topic, education of children. The gist of the conversation was that parents have limited control over their children and can "only" introduce them to different experiences. I criticized the use of the word "only" because a good introduction is not a trivial thing, it requires a lot of thought on the parent's part. The material has to match the child's ability to grasp and find it interesting enough to pursue it further on his/her own.

Examples:

• Experience foreign languages via computer games, songs, movies and trips abroad.
• Musical instruments. I played the guitar and keyboard in my younger years and know that playing an instrument is not a matter of talent but a matter of persistent work, i.e. not suitable for most young kids. Talent comes into play when one strives to be a professional.
• Math, history, politics... All of these are extremely interesting but the wrong teacher can cause the child to be disgusted and block them out completely. I know people who refused math or history until they were 30 years old and found out that these were actually quite fun.
• Sports, dance and other physical activities: See The Inner Game of Tennis

The first rule of a good introduction for any topic: What good is this for, where is it used? Keep in mind that even if your child is not immediately enthusiastic about the subject, he/she will have it imprinted in his/her mind and when the time comes, will have less difficulty in making that first attempt (which is usually the hardest psychlogically). Example: As a young child I observed how my father (a primary school teacher) was writing with beautiful hand writing his normally boring daily classroom plans. To this day, I like to write nicely. We don't want our kids to realize how everything is so mind blowing until they are adults, do we?

Self learning update for August 2018

Summer vacation is a good time for self learning. I reviewed the following:

• Bayes' Theorem
• Kalman Filter
• Graph Search, Tree Traversal, Binary Search Tree
• AC and DC
• Using Arduino Pro Micro, controlling addressable LEDs
• Proof of Pythagoras Theorem

Tavsiye: Pınar Sütkrem

Birkaç hafta önce tanıştığım ve çok beğendiğim Pınar Sütkrem ürününü bal-kaymak kombinasyonunu seven herkese tavsiye ediyorum. Bir oturuşta yarısını bitiriyorum.

Writing source code comments

A conversation I had today with a junior engineer:

Me: This code is quite complex, why didn't you write explanatory comments?
He: Since it is very complex, writing comments was difficult.

Lesson learnt: Review code of juniors as often as possible, don't let them off the hook (!)

Solve Netbeans src folder missing problem

Recently we updated from Netbeans 8.0.2 to 8.2. One of my colleagues could not see the src folder in Netbeans. What you need to do is to right click on project, open properties, go to Sources, under Source Package Folders, add the src and test folders there.

Important Netbeans folders on Windows:

• user/AppData/Local/Netbeans/Cache: Sometimes Netbeans shows error icons on files event tough there is no error. Delete the cache folder.
• user/AppData/Roaming/Netbeans

Building a NetBeans project on a computer without Netbeans

When building a NetBeans project with ant on a computer without Netbeans, you have to pay attention to the following:

• Be sure to have correct versions of org-netbeans-modules-java-j2seproject-copylibstask.jar. For example, if you are building with JDK6 but the copylibstask.jar is built with JDK7, you will get "Unsupported major.minor version 51.0" error.
• Reference copylibstask.jar in ant.bat runAntNoClasspath section as follows: "-Dlibs.CopyLibs.classpath=%ANT_HOME%\extra\org-netbeans-modules-java-j2seproject-copylibstask.jar". If you don't do this in ant.bat and you have Netbeans installed, ant will look for copylibstask.jar in Netbeans install dir which means your build dependens on having Netbeans installed.
• If you get "Class not found: javac1.8", it means you ant version is too old and not compatible with JDK8.
• If your modules have forms, you need a copy of swing-layout jar file. This jar includes the org.jdesktop.layout.GroupLayout class.
• You have to add dependencies to build.xml. These dependencies are specified in nbproject/project.properties in the javac.classpath section. Example:

• If you want to embed libraries into main jar:

Solving the Monty Hall problem using Bayesian inference

The Monty Hall problem is one of my favorites. I came across How to solve the Monty Hall problem using Bayesian inference, which used my other favorite topic, Bayesian inference. My only criticism of the post is that P(E) is a little confusing. The following formulation with separate P(B*) and P(C*) notation suits me better:

P(B*) = Probability that Monty will open door B (given that the user has chosen door A) = 1/2

P(C*) = Probability that Monty will open door C (given that the user has chosen door A) = 1/2

P(C*/B) = Probability that Monty will open door C (given that the user has chosen door A and car is behind door B) = 1/1

P(C*/A) = Probability that Monty will open door C (given that the user has chosen door A and car is behind door A) = 1/2

P(B*/C) = Probability that Monty will open door B (given that the user has chosen door A and car is behind door C) = 1/1

P(A/B*) = Probability that the car is behind door A, given that Monty has opened door B = P(B*/A) * P(A) / P(B*) = 1/2 * 1/3 * 1/2 = 1/3

P(C/B*) = Probability that the car is behind door C, given that Monty has opened door B = P(B*/C) * P(C) / P(B*) = 1/1 * 1/3 * 1/2 = 2/3

P(A/C*) = Probability that the car is behind door A, given that Monty has opened door C = P(C*/A) * P(A) / P(C*) = 1/2 * 1/3 * 1/2 = 1/3

P(B/C*) = Probability that the car is behind door B, given that Monty has opened door C = P(C*/B) * P(B) / P(C*) = 1/1 * 1/3 * 1/2 = 2/3

As you can see, it is two times better to switch doors, i.e. if Monty has opened door B, switch to door C. If Monty has opened door C, switch to door B. Note that if Monty was truly random, then the probabilities of P(C*/B) and P(B*/C) would not be 1/1 but 1/2 which in turn would cause P(C/B*) and P(B/C*) to be 1/3 and switching doors would make no difference. The probabilities change due to Monty knowing where the car is, i.e. by opening a door, Monty adds non-random information (evidence) into the system and we have to update the initial probabilities which were valid before this information.

Interesting variation of this problem [Think Bayes, p.10]: What if Monty always chooses door B when B does not have a car behind?

All your Bayes are belong to us!

The Bayesian Trap

A friendly introduction to Bayes Theorem and Hidden Markov Models

Behaviour before intellect

A friend of mine who has a young child asked me whether I had any curriculum in mind for teaching a child. I told him that for my 6 year old, my focus was on making sure that he was behaving properly, i.e. basic discipline, not on any intellectual subjects like mathematics. Behaviour is the foundation on which everything else is built.

Difference between hub and switch

You can listen to traffic/packets by connecting to a hub but not a switch.

The Difference Between a Router, Switch and Hub: "In a hub, a frame is passed along or "broadcast" to every one of its ports. It doesn't matter that the frame is only destined for one port... A switch, however, keeps a record of the MAC addresses of all the devices connected to it. With this information, a switch can identify which system is sitting on which port. So when a frame is received, it knows exactly which port to send it to..."

How do i capture all traffic on a switch: "A switch will never forward 'other' traffic (traffic that is not directed to your ethernet mac address + broadcast) to your port unless you tell it to do so. So, if you did not configure a mirror port on the switch, you will only see ... your own traffic & traffic to multicast and/or broadcast addresses"

JTable background paint problem

I have a Java JTable with JFormattedTextField in it. I want to set the background color of text field to red when the value in it is not valid. Naturally, I added a focus listener to the text field and set the background color in focus lost event. During tests, I noticed that the background changes when I click on another text field in table but does not change when I press the tab key. The solution is to call JTable.repaint() after setting the background. I think the reason it works when I click on another text field is that during that click, JTable.repaint() is automatically called. Another adventure/nightmare in Java UI...

Social Media Diet

As someone whose job requires strategy development, team management, design and software development, concentration is my main asset. When left unchecked, social media consumes a lot of concentration. My social media habits are as follows:

• I do not watch TV. I rarely watch any movies since I have already watched the best ones.
• My main sources of information are my YouTube subscriptions. I spend at least 1 hour a day watching videos of those channels. They are related to technology, lifestyle and business, with the occasional cat video sprinkled in between.
• I listen to podcasts or ebooks while commuting.
• All notifications on my mobile are turned off.
• On my mobile, do not disturb is automatically turned on after 22:00.
• Most Whatsapp groups are in silent mode.
• I don't have Facebook, Instagram, Twitter on my mobile.
• I have Facebook on my laptop. Most of my contacts on Facebook are in unfollow mode. This cleans up my Facebook page and let's me spend less than 5 minutes a day on Facebook. I use Facebook mainly as a photo album of friends, not as a source of information.
• I don't look at my mobile while eating or in WC.

These habits free up a lot of time compared to the majority of people who seem to be addicted to their mobile. I spend that time on deep thought which provides me with lots of ideas that I note in Gmail, Calendar or Drive.

Effective Jira

I am using Jira to record and track issues. After I have recorded an issue, my daily workflow is as follows: In the morning, I enter a comment about what I plan to do that day regrading theissue and how I plan to do it, including any questions I have. During the day, I update the comment with what I did and how I answered my own questions, or sometimes, how I got stuck and had to change course. In a way, I do micro-planning, micro-design and micro-implementation, all in a diary style.

The next time I work on the issue, I repeat the same workflow. This allows me to clarify the problem and help me quickly get up to speed when there is a long delay (e.g. 1 week) before I can touch the issue again. It also helps me prepare for our daily Scrum meetings. If it is a long running issue, I can quickly see what I did along the way and be able to easily explain why it took so long, especially if it is an issue that needs input from other people and those inputs were delayed.

New web application

I developed a new web application and deployed it on my VPS. It has a HTML + Javascript frontend and a Nodejs backend. The app daily checks if a certain Turkish law (4734 Kamu İhale Kanunu) has been changed and informs the people on the email list.

I used Nodemailer and Mailgun to send emails. I first tried Gmail but it blocked messages to other emails, so I switched to Mailgun.

How to follow up

When you assign a task to someone else or wait for their input, ask them when they think they will finish. If they say "I don't know", ask them when they will be able to estimate. Note that date in your personal notification system (e.g. Google Calendar). When the day comes, send them an email about the status of work and a new finish date estimate. If they don't answer your email within a day, call them or have a face-to-face meeting. Repeat, don't let them off the hook (!) Otherwise, many tasks are forgotten and drag on for months. I have personally experienced tasks where the initial estimate had been three weeks and were actually finished in three years, I kid you not!

How to get novice engineers up to speed without wasting time

Fresh engineers are usually not of much use in design work. If you involve them too early, they will mess things up, ask too many questions and waste everyone's time. The best strategy is to assign them improvement of documentation and testing of an existing system. Since they don't know much, any difficulty they have is an opportunity to improve documentation and procedures. That way they can acclimate to the work environment and be useful from day one.

Another important point is to get frequent feedback (e.g. with daily Scrum meetings) because they will retain only 50% of what they have been told and veer off course pretty quickly if not course corrected. Ask them to log in daily comments to the issue assigned to them in the issue tracking system (e.g. Jira).

Smallest negative double in Java

In Java Double.MIN_VALUE is the smallest positive double value. This different for Integer and Long classes: Integer.MIN_VALUE and Long.MIN_VALUE are the smallest negative values. If you want to use the smallest negative double value, you have to use -1*Double.MAX_VALUE.

Visual studio tools / extensions and code snippets

Tools/extensions that I use with Visual Studio:

1. ReSharper: For effective refactoring. Roslynator is a free alternative.
2. Match Margin: "...highlights text in the editor for matches of the word under the caret". Note: This can also be done by highlighting the word and pressing ctrl+f

My code snippets can be found in Gist.

How to teach a child to discover truths

Recently I was talking with a friend who has a 3 year old daughter about how to best rear a child. My own son is 5.5 years old.

Whenever my son has a question with a non-trivial answer, e.g. what resources are needed to build an umbrella in the game Don't Starve, my reaction is "let's ask the internet" and I show him that most questions have an answer when you know how to look for it. I don't want to be the father who knows everything but rather who knows how to find things out.

Of course, there are much more to discovering the truth, like asking correct questions and honing his bullshit detectors, than these simple exercises. He needs to grow up a little for that.

İlginç bir ifade

Bugün bir dokümanda gördüğüm ilginç bir ifade:

...nazik yardımlarınızı rica etmekten şeref duyarız.

Transient field initialization when using GSON

When you use GSON to deserialize a JSON string in your Java code, the class has to have a non-argument constructor in order for the transient fields to be initialized properly:


The output of the above code is as follows:

ClassWithImplicitNonArgConstructor
----------------------------------
json = {"val1":500}
value1 = 500
value3 = 3
intArray = [1, 2, 3]

ClassWithOnlyArgumentConstructor
--------------------------------
json = {"val1":1}
value3 = 0
intArray = null

ClassWithExplicitNonArgConstructor
----------------------------------
json = {"val1":1}
value3 = 3
intArray = [1, 2, 3]

Solving Processing path problem

Today I downloaded Processing 3.3.7 and when I ran processing.exe, I got the error message "Problem while trying to get the sketchbook" and nothing happened, Processing did not open.

I guess it is due to the fact that my windows documents folder is not in it's default location.

After searching the internet for half an hour, I found a solution (inspired by this and this):

1. Locate preferences.txt. On windows 7, it is in c:\Users\<userName>\AppData\Roaming\Processing\
2. Add the following line to preferences.txt: sketchbook.path.three=.
3. Save preferences.txt and run processing.

How to use engineering standards

When I mention engineering standards I either get blank stares or yawning in response. During my career spanning more than two decades, I saw that engineers usually fall between two extremes, heroic or bureaucratic:

What benefit do standards provide? They never guarantee the quality of a complex product but if applied correctly, they save the company from simple mistakes. The majority of resources are usually expended on creating the same problem and solving it again a million times. Standards also ease information exchange between engineers. When introduced to the organization for the first time, they increase short term cost but can be profitable in the long run. But never forget that "A Fool with a Tool is Still a Fool".

For high quality complex products, you need good engineers in a company that already has workflows and documentation practices in place. If your product is not complex and you don't care if your company exists for the next 5 to 10 years, you can throw away the standards and just focus on hiring good engineers.

Very few people are in the sweet spot of wise engineers who understand project needs and tailor standards accordingly. Examples of questions related to standards only wise engineers can answer:

• What is the safety level of the project? This needs a good understanding of the statement of work / technical specification prepared by the customer and affects everything that follows. It might be the difference between going bankrupt and finishing the project successfully.
• Which standards and sections of standards are applicable? This also depends on the safety level.
• Which documents can be combined?
• When a project is already close to completion and due to last minute certification concerns adherence to standards have to be demonstrated to an independent assessor: Which document should be created first? This priority is very important, because most probably, you won't be able to prepare all the documents at the time the assessor shows up on your doorstep. If you have prepared the most important docs, you might gain time and sympathy instead of being labelled as clueless morons.

Android Studio nightmares

Today I tried to build an Android Studio that was written by someone else three years ago. After some initial progress I got stuck with Error:java.util.concurrent.ExecutionException: com.android.tools.aapt2.Aapt2Exception. After hours of struggle and fruitless trials, I finally looked at the details of the Gradle stack trace and saw that the top message was Execution failed for task :app:mergeDebugResources  and the bottom message was com.android.builder.png.AaptProcess$NotifierProcessOutput.out.

My problem was due to an image file name ending with .9.png. I changed the ending to .png and the problem disappeared. Thanks Gradle for the cryptic concurrent.ExecutionException (!) This sort of unhelpful error messages is a typical theme when developing Android apps, especially when dealing with legacy code. You can get stuck for days on problems that you could not have foreseen.