I needed a family of functions f(x; a) with a single parameter to transform probabilities based on a set of heuristics. The family should include both convex and concave functions. The family should also satisfy the following constraints:

f(0; a) = 0
f(1; a) = 1
f(x; 1) ≅ x
f(0; inf) ≅ step function at 0
f(1; 0) ≅ step function at 1

Practically speaking, f(x) = x^a functions well for these constraints, but I wanted something “symmetric” for a very specific meaning of the word:

f(1-f(x)) = 1 - x

In words, the graph should be symmetric around the line x + y = 1.

Despite not having any practical need, I spent a couple hours on it, chatted with a buddy who enjoys such puzzles, and even ended up emailing an old math professor of mine looking for an answer. My buddy, the brilliant Chris Poirel, ended up coming through: use a parameterized, transformed quadrant of the circle:

f(x; a) = ((1-(1-x)^a)^(1/a)

This is the “upper left” quadrant, moved over, and parameterized by how much curve it has. Large a looks like a box, while small a (0 < a < 1) starts to look like a star. And of course

f(x; 1) = x 

I've got a system that is running constantly and producing data. Periodically this data is swept up and processed in a batch, and the processing archives off the chunk of raw data it processes, gzipping it along the way. As a result, after a period of time, there are a number of gzipped, textual data files sitting around. I was interested in batching these up into a single, larger file that I could upload to S3 or the like for later reprocessing. I sat down to write a bash script to take care of this process, but before I got through my first for loop, I wondered if there could be a better way.

I discovered that the gzip format can contain multiple compressed chunks, which are concatenated when decompressed! From Wikipedia:

[Gzip's] file format also allows for multiple such streams to be
concatenated (zipped files are simply decompressed concatenated as if
they were originally one file)

That means that the gzipped files themselves can be concatenated, and the effect is as if you had concatenated the original files and then gzipped. Check it out:

vagrant@vagrant-ubuntu-trusty-64:/tmp$ cat test1 test2 
My
name 
is
Joseph
Turner
and
I
love
pancakes
vagrant@vagrant-ubuntu-trusty-64:/tmp$ gzip test1
vagrant@vagrant-ubuntu-trusty-64:/tmp$ gzip test2
vagrant@vagrant-ubuntu-trusty-64:/tmp$ cat test1.gz test2.gz > test3.gz
vagrant@vagrant-ubuntu-trusty-64:/tmp$ gunzip test3.gz 
vagrant@vagrant-ubuntu-trusty-64:/tmp$ cat test3 
My
name 
is
Joseph
Turner
and
I
love
pancakes

One downside is that the original filenames are lost, so decompressing with the -N flag results in a file with the first chunk's filename:

vagrant@vagrant-ubuntu-trusty-64:/tmp$ cat test1.gz test2.gz > test3.gz
vagrant@vagrant-ubuntu-trusty-64:/tmp$ gunzip -N test3.gz
vagrant@vagrant-ubuntu-trusty-64:/tmp$ ls
test1  test1.gz  test2.gz

For my purpose, which would prefer the concatenated file, this was a small price to pay for avoiding the roundtrip compression. If you need to preserve original filenames, you can simply use the tar utility to combine the compressed files into a single file.

Today I found myself needing to concatenate many files to make one long file. I originally started with a simple solution:

cat /my/path/*.data

This worked fine, until there were more files than cat could handle (on my system, this was somewhere around 150k files):

/tmp/tests$ for i in `seq 150000`; do echo $i > $i; done           
/tmp/tests$ cat *                                                  
-bash: /bin/cat: Argument list too long

Next, I tried a simple loop:

# Elided: code to check for big_file.data and create/clear it if it
#   is missing
for file in `ls /my/path | egrep '.data$'`; do
    cat /my/path/$file >> big_file.data;
done                

This works, but is slow, much slower than the first solution. No need to despair: a bit of research uncovered a neat feature for the find command:

-exec command {} +
       This variant of the -exec action runs the specified command on the selected  files,
       but  the command line is built by appending each selected file name at the end; the
       total number of invocations of the command will be much less  than  the  number  of
       matched  files.   The  command line is built in much the same way that xargs builds
       its command lines.  Only one instance of `{}' is allowed within the  command.   The
       command is executed in the starting directory.

In other words, it'll batch the results for you and pass them to the command. In our example, we wind up with this:

find /my/path -maxdepth 1 -type f -name '*.data' -exec cat {} + > big_file.data

This will batch together files into reasonable chunks and call the command (cat in this case) with the chunks!

A couple years ago, I set out to teach myself about real estate investing. I read some books (this one being my favorite), talked to some smarter and more experienced people, and synthesized that information with my own experience and logic. And then got after it. I bought myself a fixer-upper to live in, rented out my condo, and bought a foreclosure (one of the dreaded $30k properties) to fix up and rent. Ultimately, my girlfriend got into grad school in North Carolina and I ended up liquidating all three, so it wasn't the slow-build, buy-and-hold investing I had set out to do. I ended up with more tax liability than I would have planned for, but in the end I made a tidy bit of money on each property.

I learned a ton along the way, but I think my (modest) success was more a function of distilling the information I absorbed into a set of what Ray Dalio calls principles: broadly applicable concepts that generalize well to a number of situations. By understanding and adhering to these principals, I was able to make money from my properties, even without the long timeline I had originally envisioned.

What follows are what I jokingly referred to as The Rules of Real Estate. When my girlfriend and I began looking at houses to start our journey, she would sometimes become enamored with a property or overly excited. I would slap on a half-smile and refer her to one of the "rules" she was violating. This banter contributed to my success, by helping reinforce these rules in my mind. Because they came about this way, they look a lot like a set of real estate-related cognitive biases: stumbling blocks we set up for ourselves in the evaluation of real estate deals. Following these principles helps you avoid common mental pitfalls so that you have a better chance of making money on any deal you do.

The Rules of Real Estate

Rule 1: Don't make decisions based on emotion

This rule is most often violated when looking at purchasing your own home, and much of the real estate "machine" pushes you toward an emotional decision. You tour properties with your agent. Maybe your agent takes you to to a few beaters first, to show off some of the "other inventory". You plod through each property, finding fault after fault, and despair ever finding a home. Then you go to a property that is well-updated, looks great, and smells like cookies, even if it is 15% above what you wanted to pay. Your agent mentions that they've already got other offers they're considering, or that she's shown this property a lot and there's been a ton of interest. You go home and have a discussion with your partner: there's not much left on the market right now, and someone is sure to scoop this one up! Let's make an offer near asking price to ensure we get it, ahead of the other offers.

You've just made a decision based on emotion, and as a result you're less likely to make money on the deal. Don't feel bad; it's really hard not to, and it requires discipline and diligence on your part. As I said above, many of the steps I just described prey on your human nature. The long day of touring homes results in decision fatigue. The dumpy properties you looked at first activate the contrast effect, making the last property look like a mansion, and the lack of inventory triggers a scarcity response. The sidenote about other offers triggers yet another scarcity heuristic, this one about time: you need to get an offer in quick! All of these factors trigger emotional responses, and put your mind in a state where it is challenging to make the best decisions.

Objectively, these emotional triggers are just that and little else. There are ugly, overpriced, terrible properties in any market. There will always be more inventory if you take your time to find a deal. And your offer should always be an objective reflection of your efforts at valuing the property yourself. So how do we put aside the inbuilt, emotional responses and engage our logical brains?

It turns out that we can't, really. The best we can do is to be informed of our shortcomings, and to insist on structuring our choices in such a way that we have time for logical reflection. The former can be accomplished through books like this one or this one (my personal favorite), or even blog posts like the one you're reading. The latter can be accomplished by carefully breaking each deal down in the same way, financially and logically, while giving oneself some reasonable amount of time for reflection. The time helps avoid the scarcity problems and the methodical deal evaluation engages the logical parts of our mind, helping to ensure we don't make decisions on emotion alone. After evaluating enough deals like this, it becomes easier and we become more resistant to the nefarious (intentional or not) traps laid for our emotional mind.

Rule 2: Don't count your chickens before they hatch

Deals fall through all the time, for all sorts of reasons. As humans, we tend to inordinately weight a loss relative to an equivalent gain. This means that the loss of a deal, particularly a deal in which we've invested ourselves emotionally, loom large. Additionally, for a deal to have even been on the table in the first place, we'd need to have made an investment, and we perceive that investment as a sunk cost. Together, these factors mean that we tend to go to lengths, sometimes inappropriate lengths, to keep a dealt together.

As an example, imagine you have a home inspection on a property. The inspection uncovers some hidden issues, and you ask for changes. The owner refuses to address the changes, and refuses to change the sale price. Often, rather than lose the deal we've already invested time (and maybe some money!) in, we make concessions and accept the property with the uncovered warts to keep the deal together.

The problem comes when these concessions change the parameters of the deal we've evaluated. If it was a good deal with no changes required, and it is no longer a good deal with the changes you'd accept, then you should walk away from the deal. More generally, don't count your chickens before they hatch: assume the deal will fall through until the day you get the title. This is true for both purchases and sales.

By distancing yourself from the deal, and assuming the deal will fall through, it becomes easier to objectively evaluate the situation as it evolves, ultimately resulting in better decisions.

Rule 3: There will always be another opportunity

This rule is a generalization of the two previous rules. It is a mindset we should try to coach ourselves into, one that helps shield us from emotional triggers and loss aversion: no matter what deal we are currently evaluating, no matter how good it seems or how it evolves, it is not the only opportunity out there. One of the implications of this rule is that our mental time scales should reflect the reality of real estate investing; in other words, many deals really only make sense in a time scale measured in years, but we tend to evaluate any given deal as though the inventory and opportunities we've looked at recently are all there is.

The reality is that there will always be another opportunity. Patience, along with an appropriate mindset, means we get the best deals not just for right now, but for the years to come and for our total investment potential. In turn, this translates into a better portfolio, which is a much better goal.

Rule 4: You make your money when you buy, not when you sell

At this point, you've prepared yourself. You've got a great mindset, you've shielded yourself emotionally, and you're aware of your loss aversion. The previous rules have you in a position to objectively evaluate your deals. To do so, however, you need to do do some homework and build your knowledge base.

One critical element is researching and internalizing deal evaluation techniques, including rules of thumb (examples for buy-and-hold rental properties: the 50% rule and the 2% rule). These rules vary considerably depending on what type of deal you're looking for (rentals, flips, etc.). I suggest reading as many books as you can find and finding a mentor who has done the type of investing you're interested in.

Another element is understanding the market you're working in. This means putting the legwork in to get a feel for neighborhoods, prices in your area, the way property values are changing, and more. If you're working with a realtor, they can offer some guidance on these elements, but nothing beats hitting the streets yourself to understand neighborhoods and how they're evolving. This is a time consuming process, but it's critical to being able to objectively evaluate deals.

If you're looking at un-renovated properties (and you probably are), and especially distressed properties, you also need to start learning how to make rough estimates for the costs of rehab. If you plan on doing some of the work yourself, start educating yourself on how to do the types of tasks you want to take on. You should also start getting references and building relationships with local contractors. A good, reliable contractor with a fair price can make the difference between a lucrative deal and a break-even or money losing scenario.

Depending on the type of deal, there are probably also other elements you need to educate yourself on. The point is, until you build your knowledge base, you are not ready to objectively evaluate deals. Why is this so important? Because you make your money when you buy, not when you sell. In other words, the only way to reliably make money is to buy properties that are objectively good deals. Do your homework, put in the legwork, and find the deals that will be good regardless of what happens in the market.

Rule 5: Always do your diligence

To stand the best chance of making your money when you buy, you need to fully understand the property you are buying. This means having all inspections necessary to uncover any hidden issues. The range of inspections can vary from property to property, but at a minimum this would include a home inspection, with follow up on any specific issues uncovered.

As an example, say you had a home inspection and the inspector noted cracks in the foundation. In some cases, these cracks are not an issue, but in others they can imply serious stress or even existing damage to a foundation element. Rather than accepting the inspector report which simply states that there are cracks, follow up with a licensed engineer to get an evaluation of the foundation. At first, follow ups like this can seem expensive, but the are far less expensive than finding out the house needs foundation work after you buy it.

The output of this diligence should be a list of known issues with reasonable estimates on the cost of and urgency of repair. This list then serves as input to the model you build as part of your deal evaluation. If you are planning to do some work on the property yourself, you may want to address many of the issues uncovered yourself. This doesn't mean you shouldn't get a reasonable estimate though: these estimates help you determine the discount on the asking price you should expect, and they provide leverage for your negotiation. In some states, you can include an "inspection contingency" - a way to back out of the contract if the inspection uncovers something you don't like. These contingencies are a great way to apply leverage to the seller of the home. They only work, though, if you're willing to do your diligence every time.

Rule 6: You can change anything about a property except its location

Issues uncovered in your inspection, ugly finishes, bad layouts, weird construction techniques - these are all issues with remedies. A good contractor can solve almost any problem with a property, though it may not be cheap. The one thing no contractor can fix, though, is a problem with the location. As a result, location should be the foremost concern when evaluating a deal.

If you've done your legwork, you should have a good general understanding of the neighborhoods and areas in which you are looking. Does the area have a lot of crime, or is it near such an area? Is there a ton of noise from nearby airports, railroads, or highways? Are busy streets dangerous for children living there? How are the schools in the area, and which are the most sought after? Are there sidewalks connecting the property to grocery stores, restaurants, bars, and other amenities? Is it a reasonable walk? Are there bike lanes? Is there public transit nearby? What service providers (internet, cable, etc.) are available in the area?

You should develop reasonable answers to all these questions as part of your evaluation. The desirability of a property, whether for resale or rental, is intimately tied to these factors, and so you should use them as a way to evaluate a deal. It helps to make a checklist as you do your evaluation to ensure you have good answers and don't overlook any aspect.

Another set of questions, though harder to answer, is about the future of the location. Is there new construction nearby that will change the property value? Is the area around the property being revamped? These sorts of questions are most easily answered by either being or working with an expert in the area. Realtors can fit this bill. Other area investors who have been tapped in for some time are also great sources of information. The future of a location, while not important to the value of the property today, can mean the difference between a solid rate of return and a huge home run.

To summarize, as part of your diligence, make sure you have a great understanding of the location of the property. It is the one thing you won't be able to change.

Rule 7: Break the other rules only when it makes sense

Rigid rules are a great way to minimize risk when starting a new venture. Like a poker player who only plays hands with good mathematical odds, rules prevent you from large downside exposure based on the unknown. However, the best poker players all use bluffing and other nonmathematical elements to maximize their game. Similarly, to get the best deals sometimes requires working outside these rules.

When does it make sense to break a rule? The answer is: when you have enough information to make an informed, confident estimate to the expected value, and the expected value far outstrips the cost. As with all expected value calculations, the risk is inversely proportional to your confidence. As you become more versed in deal evaluation and understanding of the risks and costs associated with common problems, the better your confidence can be in your estimate.

As an example, if you were a licensed contractor, you could more confidently make estimates of issues with the home that would be uncovered by a home inspection. This would make it less risky to forgo some of the diligence others might need to apply. However, because you had not as thoroughly evaluated the home as you would otherwise, the deal would need to be better to make up for the uncertainty. A more extreme example are homes bought at "courthouse steps" auctions - effectively sight unseen. To maximize the chances of making money on such deals, we need to be confident that the price reflects the uncertainty of a property that has not had any diligence performed.

One pitfall here is our own inability to estimate the true confidence of our own opinions. In other words, we as humans tend to overestimate our own competence. Keep this in mind as you look for those fantastic deals: you might not know as much as you thought! In general, developing reliable estimates of expected value isn't a task for beginners. For your first few deals, strive to adhere to the previous rules to develop some intuition and internalize the lessons. Once you're more confident, you can consider breaking the rules, but only when it makes sense.

These rules have served me well over the last years, but they are by no means comprehensive. Instead, they're more like guideposts I use. If you can follow them and ultimately incorporate them into your own thinking, they are an effective way to improve the returns on your real estate investments.

Did I miss any? Leave me a comment with your own rules for real estate investment.

After nearly a month of self-imposed dietary restriction, I've come to a conclusion. In order to build a healthy relationship with food, we need to recognize it for what it is: beyond being a necessity, food is often an indulgence. By accepting and internalizing that fact, we can rebuild the way we eat to produce the outcomes we really want - fitter appearance and better health.

Indulgences

Every luxury comes with a price tag, whether monetary or not. That price tag is often the only criterion used to evaluate the transaction. You see something you want, you check your bank account, find you can afford it, and buy it. Great right? Unfortunately, with every transaction comes second-order effects, indirect costs that can be difficult to evaluate, even if we already know about them.

That new pair of shoes costs you hours of your life. The shiny new car extends your working years, pushing retirement further into the future. The mortgage on the big house puts you and your family closer to subsistence, making it harder to recover from a loss of your job or any other financial setback. When evaluating these types of transactions, we tend to focus on the outcome (you, behind the wheel of the new car, turning every head in town) and the immediate cost (whew, $40,000 seems pricey, but if i finance it...), while ignoring these second order effects.

I'd argue that this is the definition of an indulgence: something we crave, but that comes at a disproportionate cost to our future selves, or those around us, or the environment, or anything else - again, monetary cost and impact is just an easy example.

Food as indulgence

In the context of this definition, food's inclusion in the category of indulgences is clear. The essence is well captured by the saying "once on the lips, forever on the hips": the second order effect of our lax dietary choices is obesity, metabolic disorder, heart disease, and everything else we already know about.

Food is unique in one way, though, and I think the distinction is what makes unhealthy eating so commonplace in America: food has almost no upfront cost at all. Soda, cookies, candy, fast food - these things are all abundantly available at a price point that boggles. Double Stuf Oreos, one my sweet tooth's favorites, clock in at a whopping 4200 calories per bag, with each bag costing under $3! For what is effectively pocket change, you can eat a phenomenally delicious taste explosion that contains roughly twice a full day's healthy calorie intake. I think this is part of the reason you see Hardee's and other fast food joints crammed with contractors and other laborers: after a morning of physical labor, you want the dopamine rush from eating delicious foods, and they are available at a price even the poorest among us can manage.

Coupled with the gradual onset of the second-order effects of a bad diet, these low prices put any and all food within reach of most Americans. The outcomes we're seeing are stark: adult obesity rates over 65% and increasing, with worsening rates among children as well. Perhaps worse, data suggests that poverty and obesity are closely related. Put another way, the people who will suffer the worst impact from the second-order effects of a bad diet are the most susceptible. For the most impoverished among us, life is largely devoid of luxury and food offers a beacon of comfort, perhaps the only one within reach. This thesis offers some insight into why the relationship between poverty and obesity might be causative; the psychological toll of being poor itself may encourage worse eating habits.

Though they may get the worst of it, it's not only the poor. The phrase "comfort food" perfectly captures the idea. When you're sick, or sad, or generally feel down, food offers a portal back into happiness and luxury. Though the focus thus far has been primarily focused on the way our relationship to food may be tied to widespread obesity, the very same concepts are what make food universally amazing beyond its prosaic position as a requirement to sustain life. This is precisely why we need to become more mindful of the relationship, though. If we let our fast thinking make the decision for us, we end up ignoring the knock-on effects. Beyond combating bad diet habits, an intentional relationship with food can make the joys of food that much better.