This post is divided into two parts: steps followed and FAQs.

Steps

Absolute Pre-requisites

1. Have an active account on IBKR. That means account verified and ready to buy stocks.
2. Make your fractional shares whole on INDMoney. That means either going from 1.6 qty to 1 qty or 2 qtys.
3. Keep at least USD 70 in the INDMoney wallet.

Now, here are the steps to transfer:

1. Log into IBKR.
2. In the menu, go to Tansfers (Deposit, Withdraw and Transfer History) (or whatever they are calling it now).
3. Go to Transfer Positions.
4. Select Incoming
5. Select region as United States as DriveWealth, the broker used by INDMoney, is in US.
6. Select ACATS as the transfer method. (usually the very first option and the most convenient)
7. Choose DriveWealth in the broker dropdown.
8. My account number looked like: IFSC-XXX-<Account number on the INDMoney app> (17 characters). Account number is the crucial part. One of my requests failed becuase of this. INDMoney doesn’t make it easy to find your account number. DON’T use the one shown on the app. Follow: US Stocks Tab → Manage → US Stocks Reports → US Trades Report to find your account numner.
9. Account Title and Tax Identification Number are pre-selected. Choose your Account Type. Mine was Individual.
10. Decide FULL ACATS or PARTIAL ACATS transfer. I did FULL ACATS. You can’t have any fractional shares for full transfer. Refer FAQs for more details.
11. Give authorisation to IBKR to take appropriate actions when the positions of you are transferring are not the products IBKR supports. IBKR has given details on what you are giving them the authorisation for. I selected Yes for everything.
12. Verify your details. Sign and submit.

IBKR will validate and confirm the request, submit it to the DriveWealth, receive the stocks. You will get an email from IBKR about the completion or failure.

FAQs

Q: What are the tax implications of the transfer process?
A: My research told me that transferring stocks incurs no capital gains tax, as it’s a dealer transfer, not a sale. Verify with your CA. Notwithstanding, maintain your transaction records with the earlier platform for tax filing purposes.

Q: Who is INDMoney’s broker?
A: DriveWealth

Q: Region to select for the transfer.
A: DriveWealth is a US broker. So region is United States of America.

Q: What is incoming ACATS and outgoing ACATS?
A: Since I wanted to get the securities out of DriveWealth, it is going to be an outgoing ACATS for DriveWealth and incoming ACATS for Interactive Brokers.

Q: Who initiates the ACATS (Automated Customer Account Transfer Service) transfer request?
A: Always the receiving broker. So, there is no need to interact with INDMoney unless there is some snag in the process and you need help.

Q: Is DriveWealth (INDMoney’s underlying broker) ACATS transfer enabled?
A: Yes, I confirmed this with INDMoney.

Q: Does DriveWealth charge a fee for the (outbound) transfer?
A: Yes, I was charged USD 65. I confirmed this with an INDMoney customer support request. Their explanation:

For outgoing ACATS transfers, a fee is typically applied by your U.S. broker. This charge is levied by the broker currently holding your assets to facilitate the transfer out of their system.

Q: How do I pay for the DriveWealth fee?
A: Keep at least USD 70 in the INDMoney wallet.

Q: Does IBKR charge a fee for the (inbound) transfer?
A: I was not charged anything.

Q: Is there a penalty for transfer failures due to any errors?
A: I was not charged anything. I successfully transfer in my third attempt.

Q: How long does it take to complete the transfer after submission.
A: It took 4 working days from submission to completion for me. Under standard conditions, ACATS transfers to IBKR complete in about 4–8 business days after submission.

Gemini says that some industry sources note 3–5 business days in smooth cases. 

Q: Can I trade on INDMoney during my transfer process?
A: I avoided it after I raised my transfer request. INDMoney’s response:

During the outgoing ACATS transfer, trading activity (including deposits, buys, and sells) will be temporarily restricted to ensure the process completes without errors.

Q: Can I trade on IBKR during my transfer process?
A: Yes.

Q: How does it work if the residency status has changed - INDMoney app having residency A and IBKR having residency B?
A: Many people on this reddit thread seemed to think so. I tried, and it worked for me.

Q: My address is different on INDMoney and IBKR. Will it work?
A: It worked for me. According to Gemini, the ACATS system relies on an exact match of key identifying information between the delivering account (INDmoney’s partner broker) and the receiving account (IBKR) to prevent unauthorized transfers. The key matching criteria is:

• Account Title/Registration: Your name must be identical on both accounts.
• Account Type: (e.g., Individual, Joint, etc.) must match.
• Tax ID: (e.g., Social Security Number, or other tax identification used) must match.

All three were supplied by me. So, I guess that’s enough. I think it would have mattered if my country of residence had been US, because then my Tax ID would have changed.

Q: How do I track my transfer?
A: INDMoney doesn’t tell you anything about the transfer or the status. You will not even get a notification/email after the transfer is complete. Only place to track the transfer is on IBKR. Follow the same steps that you followed to initiate the transfer. IBKR will guide to the the status window.

Q: How are fractional shares handled?
A: This caused a lot of research for me. Everyone online mentioned that you can’t transfer fractional shares. INDMoney also confirmed this:

Please note that fractional shares cannot be transferred. These may be liquidated as part of the transfer process.

So this was clear. What was not clear was: do I need to make them whole or will it only transfer the complete units and leave the fractional units in INDMoney? The latter part of INDMoney’s response gave the impression that it will handle the liquidation on it’s own. This assumption was incorrect. My request was rejected because of fractional shares.

So, as I mentioned in the pre-requisites, make your fractional shares whole or exit entirely. That is, turn 3.55 units of NVDA into 3 or 4 or 0 units or NVDA.

Q: Can I use partial transfer to avoid selling my fractional shares?
A: I was always trying for full transfer. When the request was rejected due to fractional shares, I contemplated using partial transfer to avoid exiting. I was avoiding selling for two reasons:

• I wasn’t sure if selling would lead to any tax implications;
• Some of my stocks were bought at very attractive prices. I want to protect my avg price.

After some thought, I decided to make them whole. Exited some of the fractional shares and bought some of the others (when there was a dip).

Q: Does the ACATS transfer also take care of transfering my buy/sell history, avg price and other associated history?
A: Not sure what is supposed to be transfered. What I got was only my avg price. All the past buy-sell history was gone. Since it was gone, my CAGR type metrics were also gone.

Q: What resources did you refer to understand the process?
A: Here are some links

• Paasa Blog
• Vested support
• Groww blog
• X/Twitter: 🚛 How to transfer your US Holdings out of Groww?
• Reddit Question 1
• Reddit discussion: how to handle INDMoney US investments after becoming NRI

1. PyTorch Fundamentals - Week 4
2. PyTorch Fundamentals - Week 1, 2, & 3

Now, summary of the week 4.

• Linear Regression through a custom DNN with a non-linearity
  • A nn.Module subclass that used a nn.Sequential to have the dense linear layer.

      dense = nn.Sequential(
          nn.Linear(input_dim, dense_dims[0]),
          nn.ReLU(),
          nn.Linear(*dense_dims),
          nn.ReLU(),
          nn.Linear(dense_dims[1], output_dim),
      )
    

• Learnt about nn.Sequential vs nn.Module.
  • nn.Sequential is a subclass of nn.Module.
  • It’s a convinient way to chain functions.
  • Think: replacing self.l1(self.l2(self.l3(x))) with self.layer123(x) where layer123 is a nn.Sequential of l1, l2, and l1.
  • Also read: What is difference between nn.Module and nn.Sequential
• Tried three different dense layer hidden unit config with each of the below loss functions:
  1. nn.MSELoss()
  2. HuberLoss() - created last week
  3. nn.L1Loss()

  Dense layer hidden unit variants: (3, 7), (4, 8), (5, 9).

• Huber Loss threw MSE Loss and L1 Loss out of the water. (Figure from TensorBoard.)

  

• TensorBoard at work
  • I was running a remote training pipeline with tensorboard logging. This pipline dumps the logs on S3.
  • The pipeline was failing with the following error: File system scheme 's3' not implemented.
  • Solution:
    • After multiple loops of [add + commit + push + job run] found the solution: pip install tensorflow-io. This is not the end of the solution.
    • The very last comment on the tensorboard github issue gives the final trick: Add import tensorflow_io to the pipeline even if not using it anywhere.

1. PyTorch Fundamentals - Week 1, 2, & 3

Now, summary of the week 4.

• Custom Loss function: Huber Loss
  • A torch.nn.Module class having with forward() method to compute the loss.

  • Huber Loss is defined as:

    \[L_{\delta}(y, \hat{y}) = \begin{cases} \frac{1}{2}(y - \hat{y})^2 & \text{for } |y - \hat{y}| \leq \delta, \\ \delta \cdot (|y - \hat{y}| - \frac{1}{2} \delta) & \text{for } |y - \hat{y}| > \delta, \end{cases}\]

    where:

    • \(y\) is the true value,
    • \(\hat{y}\) is the predicted value,
    • \(\delta\) is a threshold parameter that controls the transition between L1 and L2 loss.
  • More details about the Huber Loss
• Some custom losses in Keras and PyTorch: Loss Function Library - Keras & PyTorch
• Used the Linear Regression model to test the custom loss.
• Error 1: RuntimeError: grad can be implicitly created only for scalar outputs
  • Reason: the forward() function was returning a tensor with lenght > 1. Got the hint from PyTorch forums.
  • Fix: returned loss.mean() instead of loss.
• Error 2: All the losses were nan: this was a genuine bug in my code.
• Implementation approach 1: Use masks (my approach)

    def forward(self, y_pred, y_true):
        error = torch.abs(y_true - y_pred)
  
        flag1 = error <= self.d
        flag2 = 1 - error
  
        l2_loss = 0.5 * error**2 * flag1
        l1_loss = self.d * (error - 0.5 * self.d) * flag2
        loss = l2_loss + l1_loss
        return loss.mean()
  

• Implementation approach 2: Use torch.where() (solution provided in TorchLeet)

    def forward(self, y_pred, y_true):
        error = torch.abs(y_true - y_pred)
  
        condition = error <= self.d
        loss = torch.where(condition, 0.5 * error**2, self.d * (error - 0.5 * self.d))
        return loss.mean()
  

• Turns out, torch.where() is the most optimised way of doing this. It is vectorised and GPU-friendly. It is also a cleaner implementation of the same logic. Masking will require extra memory and extra operations (two multiplications, and one addition).
• Used tensorboard to visualise the training results.
• Read up more on optimizer.zero_grad().
  • PyTorch accumulates gradients by default. The loss.backward() will add to the previous gradients (can be accessed by weight.grad).
  • If we don’t reset the gradients using zero_grad(), the new gradient will be a combination of the old and the newly-computed gradient. Since the old gradient was already used to update the model in the last iteration, the combined gradient will point in a different direction than the minimum (or maximum.) [ref]
• Q: When should we use zero_grad()? A: When we want gradient accumulation on purponse.

• Q: When do we want gradient accumulation on purpose? A: In the following scenarios:

  1. Large batch size with limited gpu memory. Split the batch into mini-batches. Accumulate gradients for all the mini-batches and then run optimizer.step(). Used during training on smaller GPUs.
  2. Multiple loss components before a single update. Useful for multi-task learning. Losses that require multiple passes.
  3. Parallel training. When model is split across devices -> accumulate the gradients across the micro-batches and then update parameters once.
  4. Training with noisy gradients. Accumulate over multiple steps with noisy gradients to smooth the gradients before updating.

Week 1

• Create a Linear Regression model.
  • torch.nn.Linear to define a learnable model.
  • forward() for forward pass.
  • model.parameters containing all the learned weights and also passed to the optimizer (SGD, Adam, etc.)
  • Use torch.no_grad() during inferencing.
• Log the training logs to TensorBoard. (not a part of the TorchLeet repo)
  • SummaryWriter from torch.utils.tensorboard. Tensorflow can directly use a callback inside the fit function to push all the relevant logs. The SummaryWriter gives a fine-grained control to log anything.
  • add_scalar() to log the training loss.
  • add_graph() to log the graph itself.

  • Load the Jupyter tensorboard extension so that we don’t have to leave the notebook to look at the logs and and pretty plots.

      %load_ext tensorboard
    

  • Load the tensorboard UI inside the notebook.

      %tensorboard --logdir PATH_TO_LOG_DIR
    

Week 2

• Create a Dataset
  • Dataset class from torch.utils.data.
  • Create a subclass of Dataset for my specific dataset. Added data, X and y attributes to the class.
  • Since we will iterate through the rows of this dataset, defined __len__ and __getitem__ functions. These overloaded functions enable code like len(dataset) and dataset[i], respectively.
• Dataloader

  • Dataset only defines the dataset. Dataloader from torch.utils.data creates an iterator. It also brings other capabilities like batching and shuffling. Eg:

      dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
    

  • We can run a for loop on this dataloader now.

• Good intro to the topic from PyTorch - Datasets and DataLoaders.
• Trained the Linear Regression model using the dataloader. Faced some issues due to dtype mismatches - used torch.float32 everywhere to fix it.
• The exercise only asked for a single column dataset. Played around with a dataset with multiple columns.
• Use tensorboard for all the logging.

Week 3

• Two types of activation functions – with learnable parameters and without.
• Activation function with learnable parameters will require a nn.Module subclass. It is required to do the gradient calculations using forward and backward functions and get the final trained weights.
• Created a custom activation without learnable parameters: \(\text{tanh}(x) + x\).

• Updated the Linear Regression class to have the final output go through \(\text{tanh}(x) + x\) using torch.tanh().

    return self.custom_activation(self.linear*(x))
  

• This SO answer talks about how to write a custom activation function in different scenarios: non-learnable, learnable, learnable with PyTorch functions, and learnable without PyTorch functions.
• Also learned about torch.nn.Parameter and torch.nn.Variable.
• Kept using Tensorboard for all the logging.

Next 2 weeks: Custom Loss Function (Huber Loss) and Deep Neural Network

• Push everything to a sink; and
• Visualise the data in this sink.

Objective is to do weekly reviews and take interventions if things are not BAU. Long term vision is to eventually have enough signals to give me a comprehensive understanding of myself (physical, mental, social, financial, etc).

This post is about logging calls using Tasker for Android.

Logging Phone Calls

Steps:

1. Trigger on the event “Phone Idle” (whenever the phone goes to an idle state - incoming call, missed call, and outgoing call)
2. Read the data provider content://call_log/calls to get the most recent call details
3. Format the details for my use
4. Push to an an endpoint that saves this data in a table.

This is how the logs looks like:

#call(40) +type[miss] @num[Friend Number] @name[Friend nName] +mode[phone] +add[My Location]

#call(40) +type[in] @num[Friend Number] @name[Friend nName] +mode[phone] +add[My Location]

#call(84) +type[out] @num[Unsaved Caller’s Number] @name[] +mode[phone] +add[My Location]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
Profile: Log Phone Calls
        Event: Phone Idle

    Enter Task: Call Logs

    A1: Variable Set [
        Name: %call_log_cols
        To: date, geocoded_location, countryiso, type, number, name, duration
        Structure Output (JSON, etc): On ]

    A2: SQL Query [
        Mode: URI Formatted
        File: content://call_log/calls
        Columns: %call_log_cols
        Order By: date desc
        Output Column Divider: ,
        Variable Array: %call_logs

    A3: Multiple Variables Set [
        Names: %qs_ts, %call_geocoded_location, %call_countryiso, %call_type, %call_number, %caller_name, %call_duration
        Variable Names Splitter: ,
        Values: %call_logs(1)
        Values Splitter: ,
        Structure Output (JSON, etc): On ]
        Use Global Namespace: On ]

    A4: Variable Clear [
        Name: %call_logs ]

    A5: If [ %qs_ts neq %LAST_CALLTS ]

        A15: If [ %call_type eq 1 ]

            A16: Variable Set [
                Name: %call_type
                To: in
                Structure Output (JSON, etc): On ]

        A17: Else
            If  [ %call_type eq 2 ]

            A18: Variable Set [
                Name: %call_type
                To: out
                Structure Output (JSON, etc): On ]

        A19: Else
            If  [ %call_type eq 3 ]

            A20: Variable Set [
                Name: %call_type
                To: miss
                Structure Output (JSON, etc): On ]

        A21: End If

        A22: Variable Set [
            Name: %qs_note
            To: #call(%call_duration) +type[%call_type] @num[%call_number] @name[%caller_name] +mode[phone]
            Structure Output (JSON, etc): On ]

        A23: Perform Task [
            Name: Commons: POST Note & Location
            Priority: %priority
            Local Variable Passthrough: On
            Limit Passthrough To: %qs_note, %qs_ts
            Structure Output (JSON, etc): On
            Continue Task After Error:On ]

        A24: Variable Set [
            Name: %LAST_CALLTS
            To: %qs_ts
            Structure Output (JSON, etc): On ]

    A25: End If

Logging WhatsApp Calls

You can’t read whatsapp calls from some data provider like normal phone calls. WhatsApp also doesn’t support exporting the call logs. These calls are also not available in normal phone call logs. The only method was to read WhatsApp notification logs to get the details. Here are the steps:

1. Every time WhatsApp gives a notification, run the next set of steps.
2. If the notification was of a (audio/video) call, then get the data out in the relevant variables.
3. Push to an an endpoint that saves this data in a table.

Caveats:

1. WhatsApp generates separate calls related notifications:
   • incoming audio/video call
   • missed audio/call call (after an incoming call is missed)
   • If multiple calls have piled up then a separate notifation of (2+ missed calls from …)
   • An outgoing calls just says: “calling…” –> so, no audio/video label.
2. Since this is just a call notification (incoming, outgoing), there is no call duration available

This is how the logs looks like:

#call(-1) +type[miss] @num[null] @name[Friend Name] +mode[whatsapp-video] +add[My Location]

#call(-1) +type[in] @num[null] @name[Friend Name] +mode[whatsapp-video] +add[My Location]

#call(-1) +type[out] @num[null] @name[Friend Name] +mode[whatsapp-any] +add[My Location]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
Profile: Log WhatsApp Calls
    	Event: Notification [ Owner Application:WhatsApp Title:* Text:* Subtext:* Messages:* Other Text:* Cat:* New Only:On ]

    Enter Task: WhatsApp Call Logs

    A3: If [ %evtprm7 eq call ]

        A4: Multiple Variables Set [
             Names: %qs_ts, %call_geocoded_location, %call_countryiso, %call_type_str, %call_number, %caller_name, %call_duration, %call_type,%call_mode
             Variable Names Splitter: ,
             Values: %TIMEMS,,,%evtprm3,null,%evtprm2,-1,null,any
             Values Splitter: ,
             Structure Output (JSON, etc): On ]

        A5: If [ %call_type_str ~R .*Calling.* ]

            A6: Variable Set [
                 Name: %call_type
                 To: out
                 Structure Output (JSON, etc): On ]

        A7: Else
            If  [ %call_type_str ~R .*Incoming.* ]

            A8: Variable Set [
                 Name: %call_type
                 To: in
                 Structure Output (JSON, etc): On ]

        A9: Else
            If  [ %call_type_str ~R .*Missed.* ]

            A10: Variable Set [
                  Name: %call_type
                  To: miss
                  Structure Output (JSON, etc): On ]

        A11: End If

        A12: If [ %call_type_str ~R .*voice.* ]

            A13: Variable Set [
                  Name: %call_mode
                  To: voice
                  Structure Output (JSON, etc): On ]

        A14: Else
            If  [ %call_type_str ~R .*video.* ]

            A15: Variable Set [
                  Name: %call_mode
                  To: video
                  Structure Output (JSON, etc): On ]

        A16: End If

        A17: Variable Set [
              Name: %qs_note
              To: #call(%call_duration) +type[%call_type] @num[%call_number] @name[%caller_name] +mode[whatsapp-%call_mode]
              Structure Output (JSON, etc): On ]

        A18: Flash [
              Text: %qs_note
              Continue Task Immediately: On
              Dismiss On Click: On ]

        A19: Perform Task [
              Name: Commons: POST Note & Location
              Priority: %priority
              Local Variable Passthrough: On
              Limit Passthrough To: %qs_note, %qs_ts
              Structure Output (JSON, etc): On
              Continue Task After Error:On ]

        A20: Write File [
              File: Download/wa_calls.txt
              Text: %qs_ts, %call_geocoded_location, %call_countryiso, %call_type_str, %call_number, %caller_name, %call_duration, %call_type,%call_mode
             %qs_note

              Append: On
              Add Newline: On ]

    A21: End If

Bye.

• Push everything to a sink; and
• Visualise the data in this sink.

Objective is to do weekly reviews and take interventions if things are not BAU. Long term vision is to eventually have enough signals to give me a comprehensive understanding of myself (physical, mental, social, financial, etc).

Current progress in reverse chronology:

• 2025: Logging phone calls

I have tried this multile times in various formats over the years. Here are my previous efforts in reverse chronology:

• 2023: Google Fit data sync and analysis – my most successful attempt and still in-use. Pushed me to keep things simple.
• 2021: Futter app to do the logging 3 – couldn’t manage building this along with work.
• 2021: Futter app to do the logging 2
• 2021: Futter app to do the logging 1
• 2018: Analysis of my WA chats 2 – good analysis on my chatting habits, but nothing new. Led me to work on some solo research projects.
• 2016: Analysis of my WA chats
• 2014: Gamification of Life – too much information to manage, eventually started feeling like a chore.

Bye.

Summary

• Recommendation System: candidate gen + ranking.

• A typical ranking model pipeline:

  1. Label prep
  2. Feature prep
  3. Model training
  4. Model evaluation
  5. Model deployment (with inference contract)
• Each recommendation use case (e.g.: discover page, notifications, related items, category exploration, search) will have a version of the above pipeline.

• As use cases increase, the team will need to maintain multiple such pipelines. It is time-consuming to maintain multiple pipelines and increases points of failure.

  

• Since the pipelines have the same component, we can consolidate them.

• Consolidated pipeline:

  1. Label prep for each use case separately
  2. Stratified union of all the prepared labels
  3. Feature prep (separate categorical feature representing the use case)
  4. Model training
  5. Model evaluation
  6. Model deployment (with inference contract)

  

• Label prep for each use case separately

  1. Each use case will have different ways of generating the labels.
  2. Use case context details are added as separate features.
     • Search context: search query, region
     • Similar items context: source item
  3. When the use case is search, context features specific to the similar item use case will be filled with default values.

• Union of all the prepared labels

  1. Final labelled set: a% samples from use case-1 labels + b% samples from use case-2 labels + … + z% samples from use case-n labels
  2. The proportions [a, b, …, z] come from stratification
  3. Q: How is this stratification done? Platform traffic across different use cases?
  4. Q: What are the results when these proportions are business-driven? Eg: contribution to revenue.

• Feature prep

  1. All use case specific features added to the data.
  2. If a feature is only used for use case 1 then it will contain default value for all the other use cases.
  3. Add a new categorical feature task_type to the features to inform the model about the target reco task.
• Model training happens as usual: feature vector and labels. Architecture remains the same. Optimisation remains the same.

• Model evaluation

  1. Check the appropriate eval metrics to check the model.
  2. Q: How do we judge if the model performed well for all the use cases?
  3. Q: Will it require a separate evaluation set for each use case?
  4. Q: Can there be a 2nd order Simpson’s paradox here: the consolidated model performs well, but when tried for individual use cases, its performance is low? My hunch: no.

• Model deployment (with inference contract)

  1. Deploy the same model in the respective environment made for each use case. That env will have all the specific network-related knobs: batch size, throughput, latency, caching policy, parallelism, etc.
  2. Generic API contract to support the heterogenous context (search query for search, source item for related items use case.)

• Caveats

  1. The consolidated use cases should be related (eg: ranking for movies in the search and discover page)
  2. One definition of related can be: ranking the same entities.

• Advantages

  1. Reduces maintenance costs (less code; fewer deployments)
  2. Quick model iterations to all the use cases
     • Updates (new features, architecture, etc) for one use case can be applied to other use cases.
     • If consolidated tasks are related, then new features don’t cause regression in practice.
  3. Can be extended to any related use case from offline and online POV.
  4. Cross-learning: the model potentially gains more (hidden) learning from the other tasks. Eg: having search data gives more data to the model learning for related-items task.
     • Q: Is this happening? How can we verify this? One way: Train an independent model on the use-case specific data and compare its performance with the consolidated model’s performance on the same task.
• I was confused about what to call this learning paradigm. Wikipedia says that it is multi-task learning.

Practice at my work

• The models are not merged across different tasks like relevance and search.
• Within relevance ranking tasks (discover, similar items, category exploration), have a common base ranker model.
• On top of that, we have different heuristics to make it better for that particular section.
• Advantages:
  • There is only one main model for all related tasks.
  • Keeps the heuristics logic simple and, thus, easy to maintain.
• Challenges
  • Heuristics are crude/manual/semi-automated → we may be leaving some gains on the table. There are bandit-based approaches to automating it, though.
  • It loses out on cross-learning opportunities.

A common challenge, especially in larger organizations, is that your manager may not always be fully aware of the specifics of your work, and your manager’s manager likely has even less visibility. It isn’t due to a lack of interest but rather the sheer volume of responsibilities and information they handle. Additionally, even for you, it’s hard to remember all the details beyond the highlights. I find a proactive strategy essential for such scenarios: sending regular progress digests.

These digests are concise, structured email updates that you send periodically to both direct manager and their manager. The aim is to offer a clear snapshot of your activities, their impact, and your forthcoming plans. See it as a method to keep your supervisors well-informed, especially when you lack regular direct interactions.

That’s it. That is the idea. You can be creative and apply it however you want. However you decide to do it, you will see gains.

In the next section, I list the key points I usually consider in my snapshots.

Key Elements of an Effective Progress Digest

To ensure your digests are both informative and impactful, here’s what you can include:

• Specific Task Details: Provide project specifics and relevant links to the completed/picked coding tasks. It entails a 1-sentence project description, PR links, JIRA tickets and other code artefacts.
• Data Science Related: If applicable, detail the models you’ve trained and deployed. Any A/B experiments launched and test results of the ones that concluded. Also, share the project solutioning doc here.
• Documentation Efforts: Highlight any documentation you’ve created or maintained. You can also merge this with other points.
• Impact and Results: Clearly articulate the outcomes of your tasks and their value to the team and company.
• Initiatives and Discussions: Share any new ideas you’ve put forward or discussions you’ve initiated.
• Future Plans: Outline your planned next steps.

Benefits

The effort invested in creating these digests yields substantial career benefits:

• Enhances Diligence: Summarizing your work makes you more conscious of your efforts.
• Boosts Positive Perception: You are perceived as a proactive and accomplished individual.
• Creates a Performance Record: These digests serve as valuable documentation of your work, valuable during performance reviews.
• Ensures Visibility: Even if managers don’t respond directly to each email, they will read them, which ensures they are aware of your work and its progress.
• Effective at Any Stage: While this practice is advantageous when starting a new job (or joining a new team), I have found it beneficial at any stage.

Conclusion

Actively managing your visibility is key to long-term career growth. Sending out regular progress digests ensures that your work is recognized. You also establish a record of your accomplishments and demonstrate your value. This practice requires regular work but has good returns.

PS: I learned this trick on a tech podcast many years ago. If anyone knows which podcast or episode, please share it with me, and I will link it here.

Update: 14th Jan

PS: A related idea of brag documents explained beautifully by Julia Evans. Shared on this HN comment.

• CI is an interval.
• An interval which is exepected to contain the parameter being estimated (eg: population mean.)
• Typical confidence levels are 95% and 99%.
• The confidence level of a confidence interval is called Nominal coverage (probability.)
• CI with 95% confidence: random interval which contains the parameter to be estimated 95% of the time.
• Two ways to mention a confidence level of 95%:
  • Confidence interval with \(\gamma = 0.95\); 95% confidence
  • Confidence interval with \(\alpha = 0.05\); 95% confidence: \(1-\alpha = 0.95\)

• Mathematical representation

  \[P(u(X)<\theta <v(X))=\gamma\]
  • \(\theta\) is the parameter to be estimated (eg: population mean or median).
  • \(X\) is a random variable from a probability distribution with parameter \(\theta\)
  • \(u(X)\) and \(v(X)\) are random variables containing parameter \(\theta\) with probability \(\gamma\)
  • Confidence level \(\gamma\) < 1 (but close to 1). eg: 0.95

• Mathematical representation in case of normal distribution:

  \[\text{CI} = \bar{x} \pm z^* \left(\frac{\sigma}{\sqrt{n}}\right)\]

  Where:

  • \(\bar{x}\) is the sample mean.
  • \(z^*\) is the critical value corresponding to the desired confidence level
  • \(\sigma\) is the population standard deviation.
  • \(n\) is the sample size.
  • The quantity \(\displaystyle {\sigma }_{\bar {x}}={\frac {\sigma }{\sqrt {n}}}\) is also called the standard error of the mean.
  • The 95% confidence level will correspond to the 97.5th percentile of the distribution. Reason: the probability of \(\theta\) lying outside the 95% confidence level is 5%. So, 2.5% probability on both sides (if symmetric). So the range becomes 2.5% to (95+2.5)%.
• We can calculate the critical value \(z^*\) as follows:
  • If the sample size is small (< 30) or we do not know the std dev, then we use the t-statistic (Student’s t-distribution.) The t-distribution is wider and has heavier tails than the normal distribution, reflecting the increased uncertainty in small samples. Thus, it accounts for the extra variability.
  • If the sample size is large enough to make CLT valid, we use normal distribution (Z-distribution) –> z-score. Eg: a z-score of 1.96 for a 95% confidence level.
  • As the sample size increases, both methods converge.
  • It is better to use the t-statistic.
• Ref:
  • Confidence interval wiki
  • Interpretation
  • Common misunderstandings

CI Width

• The narrower the width, the higher the confidence.
• Factors that impact the width of CI are sample size, variance/standard deviation, and confidence level.
  • Sample size high –> narrow CI
  • High variance/standard dev –> wider CI
  • Higher confidence level –> wider CI (more data will lie under a higher confidence level)

Coverage

• Coverage (probability): the probability that a confidence interval will include the true value (eg: population mean.)
• The proportion of CIs (at a particular confidence level) that contain the true value (eg: population mean.)
• 95% CI coverage: For example, if you calculate a 95% confidence interval for a population mean, you are saying that if you were to take many samples and calculate a confidence interval from each one, approximately 95% of those intervals would contain the true population mean.
• Probability matching: if coverage probability is the same as nominal coverage probability.
  • Nominal coverage = 50%
  • Coverage = 10/20 = 50% (blue CIs contain the true mean)
  • Probability matching since coverage is the same as nominal coverage.
  • Image ref
• Ref:
  • Coverage probability wiki
  • Confidence interval construction

Implementation and Explorations

Now, we will go through the above concepts in code.

1
2
3
import numpy as np
import scipy.stats as st
import matplotlib.pyplot as plt

Compute CI

We implement the t-distribution and standard normal distribution to calculate the critical value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
def confidence_interval_t(sample, confidence=0.95):
    """
    Calculate the confidence interval for the mean of a sample using the t-distribution.

    This function is appropriate when the population standard deviation is unknown and
    the sample size is small (n < 30), although it works for any sample size.

    Parameters:
    sample (numpy.ndarray): The sample data as a NumPy array.
    confidence (float): The desired confidence level (default
    is 0.95 for a 95% confidence interval).

    Returns:
    tuple: Lower and upper bounds of the confidence interval.
    """
    # Ensure the sample is a NumPy array
    sample = np.array(sample)

    sample_mean = sample.mean()
    # Use Bessel's correction (ddof=1) for sample standard deviation
    sample_std = sample.std(ddof=1)
    sample_size = len(sample)
    standard_error = sample_std / np.sqrt(sample_size)

    # Determine the critical value for the specified confidence level
    critical_value = st.t.ppf((1 + confidence) / 2, df=sample_size - 1)
    margin_of_error = critical_value * standard_error

    lower_bound = sample_mean - margin_of_error
    upper_bound = sample_mean + margin_of_error

    return lower_bound, upper_bound

We use stats.t.ppf to get the critical value using the t-distribution. We can replace that with stats.norm.ppf for the z-score.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
def confidence_interval_norm(sample, confidence=0.95):
    """
    Calculate the confidence interval for the mean of a sample using the normal
    distribution (Z-distribution).

    This function is appropriate when the population standard deviation is
    known, or when the sample size is large (n >= 30), allowing the
    Central Limit Theorem to approximate the sample mean's distribution as normal.

    Parameters:
    sample (numpy.ndarray): The sample data as a NumPy array.
    confidence (float): The desired confidence level (default
    is 0.95 for a 95% confidence interval).

    Returns:
    tuple: Lower and upper bounds of the confidence interval.
    """
    # Ensure the sample is a NumPy array
    sample = np.array(sample)

    sample_mean = sample.mean()
    # Use Bessel's correction (ddof=1) for sample standard deviation
    sample_std = sample.std(ddof=1)
    sample_size = len(sample)
    standard_error = sample_std / np.sqrt(sample_size)

    # Determine the critical value for the specified confidence level
    critical_value = st.norm.ppf((1 + confidence) / 2)
    margin_of_error = critical_value * standard_error

    lower_bound = sample_mean - margin_of_error
    upper_bound = sample_mean + margin_of_error

    return lower_bound, upper_bound

Let’s compare the results with scipy implementations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
sample = np.random.random_sample(10)

print("defined functions:")
print("tstat:\t", confidence_interval_t(sample))
print("norm:\t", confidence_interval_norm(sample))

print("scipy functions:")
interval = st.t.interval(
    confidence=0.95, df=len(sample) - 1, loc=np.mean(sample), scale=st.sem(sample)
)
print("tstat:\t", interval)

interval = st.norm.interval(confidence=0.95, loc=np.mean(sample), scale=st.sem(sample))
print("norm:\t", interval)

# defined functions:
# tstat: (0.2756144976802315, 0.7458592632198344)
# norm:	 (0.3070236240157737, 0.7144501368842922)
# scipy functions:
# tstat: (0.2756144976802315, 0.7458592632198344)
# norm:	 (0.3070236240157737, 0.7144501368842922)

It is the same. In the following sections, we will use the scipy functions.

CI T-distribution vs. CI Z Distribution

We will verify if the confidence interval converges as the sample size increases in both methods. We will try both on the samples generated using the following sampling methods:

• Uniform
• Standard normal
• Poisson

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
sample_sizes = []
sample_means = []

t_interval_95_l = []
t_interval_95_r = []

norm_interval_95_l = []
norm_interval_95_r = []

for sample_size in range(10, 100, 5):
    sample = np.random.random_sample(sample_size)
    t_interval_95 = st.t.interval(
        confidence=0.95, df=len(sample) - 1, loc=np.mean(sample), scale=st.sem(sample)
    )
    norm_interval_95 = st.norm.interval(
        confidence=0.95, loc=np.mean(sample), scale=st.sem(sample)
    )
    sample_sizes.append(sample_size)
    sample_means.append(np.mean(sample))
    t_interval_95_l.append(t_interval_95[0])
    t_interval_95_r.append(t_interval_95[1])
    norm_interval_95_l.append(norm_interval_95[0])
    norm_interval_95_r.append(norm_interval_95[1])
    # print(sample_size, t_interval_95, norm_interval_95)

fig, ax = plt.subplots(figsize=(10, 5))
_ = ax.fill_between(
    sample_sizes, t_interval_95_l, t_interval_95_r, color="b", alpha=0.4
)
_ = ax.fill_between(
    sample_sizes, norm_interval_95_l, norm_interval_95_r, color="r", alpha=0.4
)
_ = ax.set_xlabel("Sample size")
_ = ax.set_ylabel("CI upper and lower bounds")
_ = ax.set_title("Uniform Distribution")

Replace np.random.random_sample with np.random.standard_normal and np.random.poisson to get standard normal and the Poisson random samples.

Here are the results:

In each figure (click to zoom), the blue part corresponds to t-distribution-based CI, and the red part is to standard normal based CI. We can observe that:

1. t-distribution based CIs are wider than standard normal based CI.
2. As the sample size increases, both converge.

CI Width Simulations

Let’s visualise how the CI width changes with different factors: confidence level, sample size, and standard deviation or variance.

Confidence Level - \(\gamma\)

The width of CI increases as the CI level increases. Intuition: as the confidence level increases, we widen the range to get the upper and lower limits of the confidence interval.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
sample = np.random.standard_normal(size=1000)
sample_mean = np.mean(sample)
sample_sem = st.sem(sample)

cis = []
t_interval_l = []
t_interval_r = []
for ci in range(0, 100, 5):
    ci = ci * 0.01
    t_interval = st.t.interval(
        confidence=ci, df=len(sample) - 1, loc=sample_mean, scale=sample_sem
    )
    cis.append(ci)
    t_interval_l.append(t_interval[0])
    t_interval_r.append(t_interval[1])

fig, ax = plt.subplots(figsize=(10, 5))
_ = ax.fill_between(cis, t_interval_l, t_interval_r, color="g", alpha=0.4)
_ = ax.set_xlabel("CI level")
_ = ax.set_ylabel("CI upper and lower bounds")

Sample Size

The width of CI reduces with as the sample size increases. Intuition: as sample size increases, we get more confident in our normal distribution parameter estimation, and thus, the confidence interval width reduces.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
sample_sizes = []
sample_means = []

t_interval_95_l = []
t_interval_95_r = []

norm_interval_95_l = []
norm_interval_95_r = []

for sample_size in range(10, 10000, 10):
    sample = np.random.standard_normal(size=sample_size)
    t_interval_95 = st.t.interval(
        confidence=0.95, df=len(sample) - 1, loc=np.mean(sample), scale=st.sem(sample)
    )
    sample_sizes.append(sample_size)
    sample_means.append(np.mean(sample))
    t_interval_95_l.append(t_interval_95[0])
    t_interval_95_r.append(t_interval_95[1])

fig, ax = plt.subplots(figsize=(10, 5))
_ = ax.fill_between(
    sample_sizes, t_interval_95_l, t_interval_95_r, color="r", alpha=0.4
)
_ = ax.set_xlabel("Sample size")
_ = ax.set_ylabel("CI upper and lower bounds")

Standard Deviation (Variance)

As with confidence level as the variance increases, we have more dispersion in the data. That leads to a wider CI width.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
stds = []
sample_means = []

t_interval_95_l = []
t_interval_95_r = []

norm_interval_95_l = []
norm_interval_95_r = []

for std in range(0, 200, 1):
    std = std * 0.01
    sample = np.random.normal(loc=0, scale=std, size=1000)
    t_interval_95 = st.t.interval(
        confidence=0.95, df=len(sample) - 1, loc=np.mean(sample), scale=st.sem(sample)
    )
    stds.append(std)
    t_interval_95_l.append(t_interval_95[0])
    t_interval_95_r.append(t_interval_95[1])

fig, ax = plt.subplots(figsize=(10, 5))
_ = ax.fill_between(stds, t_interval_95_l, t_interval_95_r, color="r", alpha=0.4)
_ = ax.set_xlabel("Standard Deviation")
_ = ax.set_ylabel("CI upper and lower bounds")

Probability Matching

Coverage probability will not always be the same as nominal coverage probability. When it matches, we get probability matching. In the below figure, out of 100 confidence intervals, 7 CIs do not contain the true mean (black.) Thus, we get a coverage of 93%, which is not the same as 95%, hence no probability matching.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
population = np.random.standard_normal(size=100000)

ci_ids = []
t_interval_l = []
t_interval_r = []
ci_contains_true_value = []

ci_level = 0.95
for i in range(100):
    sample = np.random.choice(population, size=10000, replace=False)
    t_interval = st.t.interval(
        confidence=ci_level,
        df=len(sample) - 1,
        loc=np.mean(sample),
        scale=st.sem(sample),
    )
    ci_ids.append(i + 1)
    t_interval_l.append(t_interval[0])
    t_interval_r.append(t_interval[1])

    if t_interval[0] <= 0 <= t_interval[1]:
        ci_contains_true_value.append(1)
    else:
        ci_contains_true_value.append(0)

cols = ["g" if i == 1 else "red" for i in ci_contains_true_value]
fig, ax = plt.subplots(figsize=(10, 5))
_ = ax.vlines(ci_ids, t_interval_l, t_interval_r, color=cols, alpha=0.3)
_ = ax.axhline(0, color="black")
_ = ax.set_ylabel("CI upper and lower bounds")
_ = ax.set_xticks([])
_ = ax.set_title(
    f"Coverage = {sum(ci_contains_true_value) * 100 / len(ci_contains_true_value)}%"
    f" (Nominal Coverage = {ci_level*100}%)"
)

Conclusion

A Confidence Interval is an interval over a sample that is expected to contain the distribution parameter that we are trying to estimate (eg: mean.) That means that all CIs will not contain the mean. The sample and population mean could have the following properties:

1. It does not contain the mean
2. It contains the mean somewhere in the middle
3. It contains the mean but as an outlier

Since the confidence interval is built from this sample using normal distribution the confidence interval may not contain the mean in the 1st or the 3rd scenario. That is why we take the confidence level as 95% or more to handle the 3rd scenario (demonstrated in the simulation section).

Since narrow-width confidence intervals are better (and more reliable), we should try to

• take higher confidence levels (95% or more);
• have a bigger sample size; and
• have less variance in the data.

Outline

1. Normal & Lognormal Distributions
2. Lognormal to Normal
3. Normal to Lognormal
4. Conclusion

Normal & Lognormal Distributions

The normal distribution is also called the bell curve or Gaussian distribution. The bell height represents the mean position, and the bottom width of the bell represents the spread of values (standard deviation). Thus, the shape changes as we change mu (\(\mu\)) and sigma (\(\sigma\)). The \(\mu\) is the mean or average of the sample, and \(\sigma\) is the standard deviation. We denote a normal distribution as:

Find more details about the normal distribution on Wikipedia. Here are two ways of defining a normal distribution in Python.

• Using python stdlib

1
2
3
from statistics import NormalDist
mu, sigma = 5, .5
norm_dist = NormalDist(mu, sigma)

• Using scipy

1
2
3
import scipy.stats as stats
mu, sigma = 5, .5
norm_dist = stats.norm(mu, sigma)

We get a lognormal distribution when we apply exponentiation to the normal distribution. The result is a lopsided curve. It means that there is a longer tail on the right side, where larger values occur. We denote the lognormal distribution as follows:

Since the log of the lognormal distribution is a normal distribution, we can denote the relationship as follows:

Find more details about the lognormal distribution on Wikipedia. We define a lognormal distribution in Python as follows. The Python stdlib does not have a lognormal implementation.

1
2
3
4
import numpy as np
import scipy.stats as stats
mu, sigma = 5, .5
norm_dist = stats.lognorm(s=sigma, scale=np.exp(mu))

Note: the scipy.stats.lognorm takes mu and sigma of the underlying normal distribution from which we derive the lognormal distribution. While providing the scale parameter, we take the exponentiation of the mean of the normal distribution. I found the documentation inadequate in explaining the parameters. This SO question has answers that discuss the meaning of the parameters.

Here is how both the distributions look for the same mu (\(\mu\)) and sigma (\(\sigma\)).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import numpy as np
import scipy.stats as stats
import matplotlib.pyplot as plt


# all distributions
mu, sigma = 5, .5
norm_d1 = NormalDist(mu, sigma)
lognorm_d1 = stats.lognorm(s=sigma, scale=np.exp(mu))
lognorm_d1.mu, lognorm_d1.sigma = mu, sigma

mu, sigma = 5, 1
norm_d2 = NormalDist(mu, sigma)
lognorm_d2 = stats.lognorm(s=sigma, scale=np.exp(mu))
lognorm_d2.mu, lognorm_d2.sigma = mu, sigma

mu, sigma = 4, 0.3
norm_d3 = NormalDist(mu, sigma)
lognorm_d3 = stats.lognorm(s=sigma, scale=np.exp(mu))
lognorm_d3.mu, lognorm_d3.sigma = mu, sigma

# norm y
x = np.linspace(0, 10, 500)
norm_y1 = np.array([norm_d1.pdf(i) for i in x])
norm_y2 = np.array([norm_d2.pdf(i) for i in x])
norm_y3 = np.array([norm_d3.pdf(i) for i in x])

# lognorm y
x = np.linspace(0, 800, 500)
lognorm_y1 = np.array([lognorm_d1.pdf(i) for i in x])
lognorm_y2 = np.array([lognorm_d2.pdf(i) for i in x])
lognorm_y3 = np.array([lognorm_d3.pdf(i) for i in x])


# Set the figsize
fig1, ax1 = plt.subplots(figsize=(6, 4))
ax1.plot(x, norm_y1, label=f"mu = {norm_d1.mean}; sigma = {norm_d1.stdev}")
ax1.plot(x, norm_y2, label=f"mu = {norm_d2.mean}; sigma = {norm_d2.stdev}")
ax1.plot(x, norm_y3, label=f"mu = {norm_d3.mean}; sigma = {norm_d3.stdev}")
ax1.legend()

fig2, ax2 = plt.subplots(figsize=(6, 4))
ax2.plot(x, lognorm_y1, label=f"mu = {lognorm_d1.mu}; sigma = {lognorm_d1.sigma}")
ax2.plot(x, lognorm_y2, label=f"mu = {lognorm_d2.mu}; sigma = {lognorm_d2.sigma}")
ax2.plot(x, lognorm_y3, label=f"mu = {lognorm_d3.mu}; sigma = {lognorm_d3.sigma}")
ax2.legend()

plt.show()

fig1.savefig('norm_dist.svg', format='svg', dpi=1200, bbox_inches='tight')
fig2.savefig('lognorm_dist.svg', format='svg', dpi=1200, bbox_inches='tight')

Lognormal to Normal

As mentioned in the previous section, normal distribution is just a log of the lognormal distribution. So, if \({\displaystyle \ X\sim \operatorname {Lognormal} \left(\mu _{x},\sigma _{x}^{2} \right)}\), then \({\ \displaystyle \ln(X)\sim {\mathcal {N}}(\mu ,\sigma ^{2})}\).

Let us understand this by code.

1
2
3
4
5
6
7
8
9
import numpy as np

rng = np.random.default_rng()

mu, sigma = 5, .5
lognorm_samples = rng.lognormal(mu, sigma, 10000)
# take the log of lognorm samples to derive the normal dist.
norm_samples = np.log(lognorm_samples)
print(norm_samples.mean(), norm_samples.std())

5.005339216906491 0.4934326302969564

The parameters (mean and std) of the derived normal distribution (line 7) are the same as the original parameters we provided to the lognormal dist (line 5).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# log normal dist
fig1, ax1 = plt.subplots(figsize=(5, 3))
ax1.hist(lognorm_samples, bins=50, alpha=0.7, density=True, color="orange")

x1 = np.linspace(0, 800, 500)
lognorm_d = stats.lognorm(s=sigma, scale=np.exp(mu))
lognorm_y = np.array([lognorm_d.pdf(i) for i in x1])
ax1.plot(x1, lognorm_y, label=f"mu = {mu}; sigma = {sigma}")
ax1.legend()

# normal dist
fig2, ax2 = plt.subplots(figsize=(5, 3))
ax2.hist(norm_samples, bins=50, alpha=0.7, density=True, color="orange")

x2 = np.linspace(0, 7, 500)
norm_d = stats.norm(mu, sigma)
norm_y = np.array([norm_d.pdf(i) for i in x2])
ax2.plot(x2, norm_y, label=f"mu = {mu}; sigma = {sigma}")
ax2.legend()

plt.show()
fig1.savefig('lognorm_dist2.svg', format='svg', dpi=1200, bbox_inches='tight')
fig2.savefig('norm_dist2.svg', format='svg', dpi=1200, bbox_inches='tight')

Conclusion: to convert from a lognormal to normal, take the logarithm of the lognormal sample.

Normal to Lognormal

If the logarithm of a lognormal distribution is normally distributed, then the reverse will also be true. That is, the exponential of a normal distribution will give us a lognormal distribution. In notation, if \({\displaystyle Y\sim {\mathcal {N}}(\mu ,\sigma ^{2})}\), then \({\ \displaystyle \exp(Y)\sim \operatorname {Lognormal} \left(\mu _{x},\sigma _{x}^{2} \right)\ }\).

Let’s again understand this through code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import numpy as np
import scipy.stats as stats

rng = np.random.default_rng()

mu, sigma = 5, .5
norm_samples = rng.normal(mu, sigma, 10000)

# take the exp of norm samples to derive the lognormal dist.
lognorm_samples = np.exp(norm_samples)

# fit a lognorm distribution to get the mean and std dev
shape, loc, scale = stats.lognorm.fit(lognorm_samples)
mean, stddev = np.log(scale), shape
print(mean, stddev)

4.984256782660331 0.5067622675605842

The parameters (mean and std) of the derived lognormal distribution (line 10) are the same as the original parameters we provided to the normal dist (line 6). Note that we used the [scipy.stats.lognorm.fit] method to fit the lognorm distribution on the data. It gives us the following three parameters: loc, shape and scale. The shape is same as standard deviation. To get the mean, we have to take the logarithm of the scale. We did not have to do this when we converted the lognormal to a normal distribution (previous section) because we can directly get the params (mean and std). Read this SO answer for more details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# normal dist
fig1, ax1 = plt.subplots(figsize=(5, 3))
ax1.hist(norm_samples, bins=50, alpha=0.7, density=True, color="orange")

x1 = np.linspace(0, 7, 500)
norm_d = stats.norm(mu, sigma)
norm_y = np.array([norm_d.pdf(i) for i in x1])
ax1.plot(x1, norm_y, label=f"mu = {mu}; sigma = {sigma}")
ax1.legend()

# lognormal dist
fig2, ax2 = plt.subplots(figsize=(5, 3))
ax2.hist(lognorm_samples, bins=50, alpha=0.7, density=True, color="orange")

x2 = np.linspace(0, 800, 500)
lognorm_d = stats.lognorm(s=sigma, scale=np.exp(mu))
lognorm_y = np.array([lognorm_d.pdf(i) for i in x2])
ax2.plot(x2, lognorm_y, label=f"mu = {mu}; sigma = {sigma}")
ax2.legend()

plt.show()
fig1.savefig('norm_dist3.svg', format='svg', dpi=1200, bbox_inches='tight')
fig2.savefig('lognorm_dist3.svg', format='svg', dpi=1200, bbox_inches='tight')

Conclusion: to convert from a normal to lognormal, take exp of the normal sample.

Conclusion

We started with the Normal and Lognormal distributions and with their definition in Python. We converted each of the distributions into the other. It took me some effort to figure out how to do the conversion. With this post, I tried to demystify the confusion.

If you are interested in how other distributions look, your search is over. This SO answer has visualisations of all the distributions available in scipy.stats.

Update: 18th Jan: Someone asked me the following question on reddit.

For what purpose are you converting between normal and lognormal? The two functions share the same parameters but thats about it. ln(data) is a non-destructive transformation but the process can obscure patterns just as often as it reveals them. Certain advanced statistical tests that require a normal distribution cannot necessarily have the results applied to the lognormal data.

This stranger is correct that patterns are obscured, or rather, some other patterns come up after log transformation. Although, in my case, it did not matter.

I wanted to match the customers with the items that are within the customer spending range. The formulation was that if I have customer and outlet distributions, then I can match these distributions or get the overlap to get the match percentage. This match percentage will then be used on top of relevance scores.

Looking at the customer’s spend history, I saw that the distribution was lognormally distributed. A similar trend was observed in the restaurant’s order history. Since, computing the overlap in the production env was easier with the normal distributions, I was okay with the conversion. I will cover this in more detail in a future post.

I stayed at Zostel📍. Advice: get the ten-bed dorm. It has access to the balcony. You can see all of Ooty from the balcony. At night, the lit-up Ooty City takes your mind away from everything.

On my workations, my stay in any city has taken one of the following tracks: make friends and go with the flow or explore solo. Ooty was the city of friends.

I reached the hostel early in the morning. While I waited for my dorm bed to get cleaned up, I met Arvind and Sriram. All of us were in the same dorm. Enjoying the Ooty view from the balcony, we had an engaging conversation. It was about introduction, backgrounds, work, and protocols. Introducing protocols is out of scope for this post, but do follow the shared link. The conversation ended when my time for a work meeting got closer.

There was almost a routine to our days. I get up in the morning and start my work after breakfast in the Hostel Cafe. These guys would chill or go out to some tourist points. I would have lunch and continue working till the evening. After dinner, all three of us and others (new people came and went every other day) would sit around the bonfire and talk. Sometimes we sang and played music. I would also try to produce some sounds with my Ukulele.

One evening, wrapping up early from work, three of us went for a movie. On all my travels, I have watched movies on hostel TVs, but never once have I gone to a movie in a theatre. It was a novelty experience. Let me describe the whole scene.

The movie in focus was Blue Beetle (judge all you want 😅). The only way to book the ticket was at the theatre called Assembly Rooms📍. We reached there ten minutes early. Including us, there were only five people present. The booking window was not yet open. At the show time, the ticket guy informed us that he needed at least six people to start the show. Our strength was still five. We agreed to buy an extra ticket (each ticket was 180 bucks.) After waiting for five more minutes, he let us watch the movie. The same guy was also the movie operator and started the show. The whole theatre was ours. We watched the senseless movie from the 3rd row and made fun of the various scenes.

I had conversations with both of them on a range of topics. The topic of protocols continued. We also talked about green and sustainable energy (Arvind’s forte), their college life (they know each other from college), Peru (Arvind’s base), Ooty’s past (Arvind used to come here from Coimbatore in his childhood), quantified self, music, learning music, tradition of learning music or dance in Tamil families, and a lot more.

One by one, both of them left. It is always sad when someone leaves after you have spent time with them. It is a reality of life, but it is still sad to be left behind. Despite that, you keep going, and then it becomes normal again. You meet new people, and the cycle continues.

Ooty (and the whole of Tamil Nadu) is hard on solo travellers without transport. In every city, I rent a scooty to travel around. Ooty (read: Tamil Nadu) doesn’t allow renting vehicles. The weekend was here. I wanted to see Ooty. So, I asked for the touring taxis at the reception. The taxi guy quoted 2500 bucks for the whole day. It was time to make new friends.

I met Rajesh, who was also looking to share the taxi. Meghna and Gargee were the final two. And our impromptu travel group was ready. Although the taxi driver increased the rate to 3100 INR, it was still better for all four of us.

Rajesh is a PhD student in Geology. He was studying rocks somewhere near Ooty and decided to spend the weekend here. He is more interested in Climate Science and will join another PhD program in a few weeks.

Meghna and Gargee are childhood friends from Gwalior, Madhya Pradesh. Meghna works at an IT company in Pune. Gargee is an architect in Bangalore. She left her job and was moving to Ahmedabad to become a Landscape Architect. The reason for them coming to Ooty was to enjoy Gargee’s last trip from Bangalore.

We went to the following points on our tour.

1. Pine Forest📍
2. Lake - Sandynulla📍
3. Pykara Lake📍
4. Pykara waterfalls📍
5. 9th-mile shooting point📍

It took us till evening to cover all these points. We asked our cab driver to drop us in the city market. It was a full-moon night. We checked out some shops in the market, ate our dinner at Adyar Ananda Bhavan - A2B 🍽️ and called it a night after warming up around the bonfire.

The morning, I accompanied Meghna and Gargee to the Bus Depot. They were leaving for Wayanad. After seeing them off, I went to explore the city. I saw a watch tower. On my way, I saw scores of people entering and leaving an alleyway. My curiosity made me go into the alley, and I saw a complete change in the landscape. I was standing in a fruit and veggie market. There were multiple alleys leading to different sections of the market: fruits, veggies, meat, dry fruits, flowers, and other grocery stuff. It felt like the Diagon Alley in the Harry Potter universe.

After coming back, Mugdha and Madhvi were painting the wall art in the common area.

A little backstory. My dorm was right next to the common area. On my first day in the hostel, the staff cleared and re-painted a wall in the common room for a new wall art. Mugdha came a few days later to pencil the outline. The outline consisted of the elements of Ooty: coffee plantation, toy train, toda tribe, rose garden, and pine forest. During the weekend, Madhvi arrived. Both of them started colouring the sketch. We became friends, and I started my apprenticeship under them. Soon it became a group of seven: Mugdha, Madhavi, Anshul, Sumit, Gautam, Vani, and me.

So, I started helping them with colouring after returning from the market. The last time I held a paintbrush was in the 9th grade, more than 14 years ago. I coloured different shades of roses, coffee beans, leaves, and grasslands. I enjoyed filling up those shapes using a paintbrush. I first added a colour coat without worrying about the brush strokes. Later, I painted over it to make it consistent. I aligned the strokes with the outline to make it coherent. I experimented with multiple ways of moving my brush. Slowly, I could achieve the same effect in fewer steps and less paint. It was a very calming activity. I used to leave more delicate stuff for my teachers. By the end, both of my teachers were proud of my work. 😁

Along the way, all of us talked extensively. Both Mugdha and Madhvi are from Mumbai and friends from college. Mugdha is into art, and Madhvi likes textile design. I learnt how students learn in art schools in India. Madhvi had been working in Design Thinking for school kids. She is going to get into textile design next. Mugdha is going to experiment more with painting and colours. One morning, Mugdha introduced me to Aahatein by Agnee. (This song was at the top of my Spotify Wrapped this year.) We discussed similar songs. Madhvi showed her skills on my Ukulele. I also got to know about the Kochi-Muziris Biennale. I witnessed the traces of previous iterations of Biennale when I went to Kochi a few weeks later.

Anshul, who works on APIs for clients, was trying to explain to Madhvi and Mugdha what he does. While trying to help him explain, we learnt that many women, like Madhvi and Mugdha, enjoy having nerdy talks and would date such men. Then the topic moved to interesting or weird dates many of us have experienced.

Gautam played a ten-minute movie called Zima Blue from the Love, Death & Robots animation series. Gautam is a volunteer teacher in a nearby village. He is visiting India for a few months, following which he’ll return to the US. Sumit was on a road trip on his KTM and headed back home to Pune from Ooty. We talked about many of his road trips. Vani was on a weekend trip from Bangalore.

I enjoyed painting so much that I extended my stay by a few more days. I, unfortunately, had to say goodbye to everyone before it was complete.

From the hostel, I took the bus to Conoor. And from Conoor📍, I took the toy train to Mettupalayam📍.

Fortunately, I got the window seat assigned to me. The train - running on steam - passed through multiple bridges, coffee plantations, and dark tunnels opening up to beautiful valley views. The train stops at two railway stations along the way. Both the stations had an old vibe to them. On the second stop, they also fueled up the engine with water.

At the end of this journey, I took another bus to Coimbatore📍 and reached my hotel (no hostels in Coimbatore ☹️). I met with two people here: Arvind and Guhan. Arvind, whom I had met at the beginning of this post, returned to Coimbatore after leaving Ooty. Guhan is a friend I made in Hampi earlier this year. The story of Coimbatore will continue in another post.

Will you display it as a table? Not everyone can grok it. It will also take time to walk people through the table.

You will format your table with colours (conditional formatting) to show peaks and bottoms. That will work. However, it will become dense as number of rows increase. Furthermore, this workflow involves exporting data from your respective data system (database or data lake) and importing it to Excel/Google Sheets. Thus, it is not feasible in all situations. One of those situations is what I faced.

I was doing an analysis of customer data at work. I wanted to see the distribution of cuisines in two subsequent orders. For example, the customer ordered Chinese food followed by South Indian in the next order. Because sequence matters for my analysis, Chinese to South Indian and South Indian to Chinese would be two separate rows. As you can imagine, a significant part of the GroupBy output contained these redundant pairs. It was difficult to derive any insights from it.

Bipartite Graphs to the Rescue

Fortunately for me, I was able to recall the bipartite graphs. Bipartite graphs model the relationship between two classes of objects. For example, think about the relationship between owners and their cars. An owner can own ore or more cars. An owner can not own other owners. Similarly, a car can not own other cars. A bipartite graph will only show a relationship between a vehicle and its owner (two different classes of objects).

It was perfect for my visualisation problem at hand!

However, to generate a presentable graph turned out to be slightly roundabout. This article is to document the process for my future self.

The Process

As expected, the NetworkX Python library had all the utilities available. The steps are as follows:

1. Get data
2. Define a networkx Graph.
3. Use bipartite_layout() to define the layout for a bipartite graph.
4. Draw the graph using draw().

There are more minor steps involved that we will cover during the deep dive. Since NetworkX plays well with the Matplotlib library, we have all the Matplotlib utilities available to us.

I will visualise the age-wise top causes of death according to WHO.

We start with the necessary imports.

1
2
3
4
5
import random
import pandas as pd
import networkx as nx

from matplotlib import pyplot as plt

We have to pre-process the data for the viz.

1
2
3
4
5
6
7
8
9
10
11
data = pd.read_csv("male.csv").set_index("cod").T
data.columns = ["cod_"+i for i in data.columns]
data = data.rename_axis('age_group').reset_index(drop=False)
data = pd.wide_to_long(
    data, stubnames="cod", i=['age_group'], j="cause", sep='_', suffix=r'[\w ,]+'
)
data.columns = ["percent"]
data = data.reset_index(drop=False)
data["percent"] = data["percent"].str[:-1].astype(float)/100
data = data[data.cause != "All Causes"]
data.head(2)

The data is ready. I wanted all the edges with the same start in the same colour. So I added an integer corresponding to each class using the below code. We will use this column to get a random colour for each label with a colour map.

1
2
3
4
# colors
node_dict = dict([(j, i) for i, j in enumerate(data['age_group'].unique())])
data["node_color"] = data["age_group"].apply(lambda x: node_dict[x])
data.head(2)

I am loading the data and converting the wide to the long format for NetworkX. Next, we define our graph using this data.

1
2
3
4
5
edges = [tuple(x) for x in data[['age_group', 'cause']].values.tolist()]
B = nx.Graph()
B.add_nodes_from(data['age_group'].unique(), bipartite=0)
B.add_nodes_from(data['cause'].unique(), bipartite=1)
B.add_edges_from(edges)

Below is how we visualise the graph.

1
2
3
4
5
6
7
8
9
10
11
# matplotlib variables
fig, ax = plt.subplots()
fig.set_size_inches(9, 6)

# First specify the nodes we want on left or top
# create a bipartite layout
left_or_top = data['age_group'].unique()[::-1]
pos = nx.bipartite_layout(B, left_or_top, scale=10)

# Pass that layout to nx.draw
nx.draw(B, pos, node_color='#A0CBE2', edge_color="white", width=1)

We define Matplotlib variables. Use bipartite_layout to get the required layout and draw the graph. Note that, without edge_color="white", we can stop at this step. We will get equal width, constant colour edges and nodes. The next few steps will fix the presentation aspect of the plot.

We colour the edges first.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# define random color map - https://stackoverflow.com/a/68459848/2650427
colors_ = lambda n: list(
    map(lambda i: "#" + "%06x" % random.randint(0, 0xFFFFFF), range(n)))
colors = colors_(len(data.age_group.unique()))

# draw each edge
edge_width_dict = (
    data[['age_group', "cause", "percent"]]
    .set_index(['age_group', "cause"])
)
for node in data[['age_group', "node_color"]].drop_duplicates().values:
    edges = B.edges([node[0]])
    color = colors[node[1]]
    edge_widths = [edge_width_dict.loc[i]["percent"] for i in edges]
    nx.draw_networkx_edges(
        B,
        pos,
        edgelist=edges,
        width=edge_widths,
        edge_color=color,
    )

We iterate through all the starting nodes and their corresponding colours. We get each point and its edges and colour them the same but vary their width according to the percent column.

Last configuration is the node labels and their alignment. Without this segment, all the node labels would be centre-aligned. A long string is truncated in the viz. I want to point out that neither the documentation nor Stack Overflow could help me here. My saviour was ChatGPT. It gave me a working example using draw_networkx_labels() that I modified as below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# left node labels alignment
for node_name in data['age_group'].drop_duplicates().values:
    node = {node_name: node_name}
    node_pos = {node_name: pos[node_name]}
    label_pos = nx.draw_networkx_labels(
        B, node_pos, labels=node, font_size=10,
        horizontalalignment='left',
        verticalalignment="bottom"
    )

# right node labels alignment
for node_name in data['cause'].drop_duplicates().values:
    node = {node_name: node_name}
    node_pos = {node_name: pos[node_name]}
    label_pos = nx.draw_networkx_labels(
        B, node_pos, labels=node, font_size=10,
        horizontalalignment='right',
        verticalalignment="bottom"
    )

plt.show()

Our Beautiful Plots

Time to see the results.

Male children mostly die due to Infectious and parasitic diseases, Respiratory infections, Maternal conditions, Neonatal conditions, and Nutritional deficiencies. Most teen and youth deaths (15-29 years in age) happen due to injuries. As men get old, serious ailments (Birth ailments, Cancer, Cardiovascular, Respiratory, and others) become more pronounced causes of death.

Females follow a similar distribution. One notable difference is that relatively few women die due to injuries. Is that the reason women live longer than men?

The plots effectively showed the common diseases for each age group. Of course, this plot only gives a summary. And the summary is what we wanted from this viz.

Shortcomings

The plots were 90% there. Unfortunately, there are a few flaws.

While it provides me with a summary, it does not tell me the strength of the relationship. In that aspect, it is similar to pie charts. And the internet is filled with articles about why pie charts are unhelpful plots.

Another issue is the random colour and edge width assigned to each edge. A node may be yellowish-green in colour. Even if the edge width is relatively higher, the edge will still not be prominent. I re-ran my code to get the version with the right colours. We could solve this by hand-selecting the colours and tuning the edge widths with a constant factor.

Conclusion

We wanted a summary visualisation of our GroupBy (or pivot table) output. To achieve that, we converted it into a bipartite graph and rendered it using Matplotlib.

There are flaws in this visualisation. The strength of the relationship is not apparent. Additionally, edge colour and widths need tuning to make the strong relationships prominent. Fixing these issues is a future work.

I have a JSON column in my DataFrame.

• The JSON is in string format.
• It is a nested JSON.
• It is a large string.
• I do not know the schema and want to avoid defining it manually.
• All the JSONs follow the same schema definition.

I need to format it as a JSON object (struct) to extract anything out of it. How do I convert it into a struct?

Solution

Here is the solution if you are short on time. In the next section, I discuss it in more detail.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Spark 3.2.1 | Scala 2.12
import pyspark.sql.functions as F

# Sample json we will work with.
sample_json = """
{
  "lvl1":  {
    "lvl2a": {
      "lvl3a":   {
        "lvl4a": "random_data",
        "lvl4b": "random_data"
      }
    },
    "lvl2b":   {
      "lvl3a":   {
        "lvl4a": "ramdom_data"
      },
      "lvl3b":  [
        {"lvl4a": "random_data"},
        {"lvl4b": "random_data"}
      ]
    }
  }
}
"""

# Spark dataframe with json column
df = spark.createDataFrame([(sample_json,)]*4, ["json_data"])

# determine the schema
json_schema = F.schema_of_json(df.select(F.col("json_data")).first()[0])

# converting json to struct
df = df.withColumn("json_data_struct", F.from_json("json_data", json_schema))

Details

We will use pyspark.sql.functions.schema_of_json to do our dirty work of determining the schema.

Just like any other column-based function, I expected this function to work on a column. So I tried this as below:

df = df.withColumn("sch", F.schema_of_json(F.col("json_data")))

It threw the below error:

AnalysisException: cannot resolve 'schema_of_json(json_data)' due to data type mismatch: The input json should be a foldable string expression and not null; however, got json_data.;
...

I did not know what is a foldable string. The data type of the json_data column was a string. The ChatGPT also suggested the same way of using this function. :)

The documentation and multiple Stack Overflow answers [1, 2, 3] helped me reach an explanation.

The schema_of_json needs a single string instead of a column. So I extracted one JSON string from the column and passed it to the function. This is how I did it:

1
2
json_string = df.select(F.col("json_data")).first()[0]
json_schema = F.schema_of_json(json_string)

The end.

I like self-tracking. I want to track my productivity and find potential improvements. What gets tracked also gets measured. My interest in the Quantified self has evolved.

I started with Gamification of Life, where I assigned points to everything I did. It got too overwhelming after a year.

I lack at maintaining relationships with friends and family. For insights, I analysed my chats’ metadata - Chatting Up and Chatting Up - Part II. It was interesting to see how my interactions changed over time with friends. I also wanted to play with the chat content, but NLP capabilities at that time weren’t enough to deal with Hinglish text.

I track my call logs and someday would like to analyse them.

Two years back, I even started making an app to track anything. You can read more about the app here:

• Minutes - A Quantified Self App for Myself
• Minutes - Building the Settings Page
• Minutes - Building the History Page

Eventually, other commitments and curiosities caught up, and I couldn’t finish it. 😅

I also track my spending and habits. I am sure I have skipped a few more.

Tracking Fitness

Fitness was another frontier where I tried many things. Tracking was always enabled using Google Fit and Maps, but I did not do anything with the data. Google Fit analytics on the app was helpful, but I wanted more. Time was difficult to find between work, travel, and other interests. I wanted something quick and easy to develop/maintain.

Introducing: Do More With Less (DMWL). It is a trend at work where you identify and prioritize the tasks that are quick to execute with good ROI.

I googled previous work in this direction. I found this medium article that pointed to this more helpful article doing what I wanted - Export Google Fit Daily Steps, Weight and Distance to a Google Sheet[code from the blog is available here: google-fit-to-sheets/Code.gs]. It made almost everything straightforward - setting up the app, auth, pulling and formatting data. Chat GPT complemented my lack of knowledge of Javascript to write code in Google Sheets.

Update: 19th July

I had to do set up the credentials multiple times now. Add the steps here so quick reference. In depth instructions are on apps-script-oauth2/README.md.

1. Open script editor by going to Extensions > Apps Scrip. It will open a new apps script project.
2. Name the project. Click the + in the Libraries section. In the Add a Library dialogue, add 1B7FSrk5Zi6L1rSxxTDgDEUsPzlukDsi4KGuTMorsTQHhGBzBkMun4iDF as the Script ID. This will find the Google OAuth2 Lib. Select the latest version and save.
3. Go to Project Properties from the file menu and make a note of the Script ID. This is the ID for our new project. We will need it later.
4. Open the Google API Console.
5. Create a new project and name it.
6. Go to Enable APIs and Services and find the Fitness API.
7. Go to Keys and create an OAuth Client ID. While creating the consent screen, only add the product name. Select “Web Application” in the application type. In the redirect URL add https://script.google.com/macros/d/{SCRIPTID}/usercallback and replace the {SCRIPTID} with the Script ID copied in step 3. Note down the client id and client secret created at the end. We will use these in our script. [You can read more on Setting up OAuth 2.0.]

Update: 28th July

The script stops working after a few days. This has happened twice with me. I get the following error:

Error: Access not granted or expired.
Service_.getAccessToken
@ Service.gs:518

I still haven’t found the solution, but I have a few leads:

1. I hypothesise that it is related to oauth2 details being stored in the Properties Service. The token is empty when I print the Properties.
2. This Properties Store has an expiry (likely 1 hour). I couldn’t find a way to update the oauth2 details before the expiration. Tried multiple ways after deleting and setting. This SO answer didn’t help either.

End of the Update

Code

In this section, I will discuss the coding involved. You can skip to the next section for the dashboard.

Here is how the code flow is:

1. Get today’s date.
2. Get all the specified metrics for the date. I care about the following specific events: step count, weight, heart points, and all logged activities.
   • Step counts: com.google.step_count.delta
   • Weight: com.google.heart_minutes
   • Heart Points: com.google.weight.summary
   • All logged activities: com.google.activity.segment
3. Set the precision of all the numbers and impute null values with zero.
4. Get the spreadsheet object using getActiveSpreadsheet() and append the data on the last empty row using getLastRow().
5. Copy the cell formatting of all the cells from the row before using the copyTo(destination, options) function.
6. Copy the rolling avg. formulae from the row before, again using the copyTo() function.

Google Fit API

It is dense! Extending it to my signals required scouring over the docs and multiple SO answers.

The first helpful link was: Users.dataSources: list. It gave me all the data points I can ask for from the Fit API. The next challenge was discovering the schema and what different fields meant. After several hit-n-trials, I found the Activity Types page. It gave me the required ID for each activity.

Update: 28th July: I stumbled upon the guide to the REST API of Fit.

I hope you will find these links useful.

Next Features

There are two immediate hurdles I need to cross.

I have to authorize the app every day to call the API. A quick search told me that the token expires after an hour. I have to use a parameter called expires_in to refresh the token. Unfortunately, I could not figure out how to use it.

Similarly, I have to call the function daily (by pressing a button from the menu). I can automate it through the time-driven (clock) trigger [ref: triggers]. The problem is the token expiration. The trigger will fail the next day because the token is stale.

Fitness Dashboard

Time for the final results.

Google Fit gives you a Heart Point (HP) for each minute of activity you do. Here is how mine looks. My heart points mainly include walking, working out, and a little bit of swimming.

The red line is an aggregated line to make it easy to see the trend.

The yellow-shaded regions highlight the days I was workationing. During these days, my heart points rarely reached zero. Whereas in the non-shaded periods, I frequently hit zero. Those zeroes are my two rest days after five days of working out.

My heart points are also cyclic in nature. Whenever I am home, my only regular activity is exercising with a two-day break every week. Walking becomes an occasional affair.

Let’s look at these heart points more closely.

Can you notice the complementary nature of the two graphs? First, look at the red line and then focus on blue.

During the travel period, my HP fluctuated because of the crazy number of daily steps (a proxy for walking) and irregular/short workouts. And when I am at home, I go crazy with my exercise. During May, a few things at home led to more walking and small and irregular workouts.

That’s the end of my DMWL version of the Fitness dashboard.

Next Steps

The first stage of my dashboard is complete. I will iteratively update it to get more out of it. I discussed the tech improvements in the coding section. As a part of my fitness tracking journey, here is what I want to do.

I want to come up with some standards or thresholds for myself. It could mean saying something like reaching 30-40 HP daily or working out for fifty minutes five days a week. I do not know what these standards will look like.

I want to use this dashboard to motivate myself. It can be a tracker for fitness-related habits. It should all be automated. Consequently, I want this dashboard to help me inculcate new fitness-related habits. Towards this vision, the next step is to add more activities to the mix, namely meditation, cycling, and swimming.

I live a healthy lifestyle. Another goal of this dashboard is to observe how different decisions in my life impact my health and productivity. That will help me course correct. Some directions are:

• What is the impact of my diet on my weight? For example, when does a high-calorie diet typically reflect in my physique? The current heuristic-based answer is two weeks but needs validation from data.
• Sleeping hour vs the workout efficiency the next day. Does routine matter? If yes, how and where?
• How does my fitness change during my travels?
• Does fitness impact my productivity?

There are more, but these are the important ones in my mind.

Update: 19th July: I added meditation and cycling under the activities section. In the nutrition section, I added calories burnt and water intake. Calories burnt is Fit’s approximation. The water intake is tracked manually in the app like the activities.

Good intentions never work, you need good mechanisms to make anything happen - Jeff Bezos.

A mechanism is a process where

1. You create a tool;
2. Drive adoption of the tool;
3. Inspect to correct course.

Read more - Building mechanisms.

I have observed that mechanisms work more efficiently than good intentions. Thus, I always try to convert good intentions into mechanisms. Looking at how Data Science projects are not as streamlined as Software Engineering projects, I searched for guidance on managing them better.

The following are my notes from Mechanisms for Effective Machine Learning Projects and related articles. These mechanisms apply to any data science project (work or hobby). Note that these mechanisms are mental tools. You have to make them a habit for mental tools to work.

Meta-Checklist for Projects

• 1-pager describing a map to the destination (1-7 days)
  • Intent or why. Quantify the problem.
  • Desired outcome. Business metric.
  • Deliverable. No need for it to be detailed.
  • Constraints. How not to solve the problem.
• Timebox the project. Based on the timeline, design a solution that fits. Ref: timebox section.
• Literature review
  • It does not have to be exhaustive.
  • Quickly identify approaches that have worked and build on them.
  • Refer to the lit. review section.
• Reviews
  • Schedule once you have the results from the initial experiments.
  • It helps with catching blindspots or critical errors.
  • Focus points
    • Input data and features
    • Offline evaluation
    • Room for improvements.
• Set up the work environment. Read: How to Set Up a Python Project For Automation and Collaboration
• Consistent documentation during the project.
  • Document whatever is not in the code.
  • Create documentation like an applied research paper: motivation, lit review, data, methodology, results, and next steps.
  • It helps with replication.
  • Read more here: Why You Need to Follow Up After Your Data Science Project.
• Have informal stand-ups with the team:

  Share unusual findings; Discuss ideas; Get help on bugs; Ask for reviews; etc.

• Regular stakeholder communication
  • Check in regularly with them.
  • It ensures that the deliverable aligns with the overall goals.
  • It is also a source of feedback and clever suggestions.
• Read more here:
  • What I Do Before a Data Science Project to Ensure Success
  • What I Do During A Data Science Project To Deliver Success

Timeboxing Projects

• It makes you focus on the most crucial tasks.
• Timebox: stretch goals wrt the project.
• Estimate: Upper bound of effort needed.
• An estimate to go from timebox to estimate: multiply by 1.5 - 3.0.
• Most aggressive timebox: halve the time spent on a similar project. Create an MVP. Quick iteration cycles. Intense.
• Comfortable-yet-challenging timebox: reduce the time by 10-20%. Good default.
• Standard timebox: for open-ended projects. 2 weeks lit. review, 4-8 weeks for prototype building, and 3-6 months for production.

Executing Projects

Mechanism to execute projects with high confidence.

• Pilot and copilot for each project.
• Pilot: main project owner.
  • Responsible for success/failure
  • Own and delegate as required.
• Copilot: helps the pilot stay on track, identify critical flaws, and call out blindspots.
  • Periodic check-ins
  • Reviews document drafts and prototypes
  • Mandatory code reviewer
• Copilot has (more) experience in the problem space.
• Copilot spends 10% of the pilot’s effort.

Literature Review

• Always start the project with a literature review.
• Read papers relevant to the problem.
• Start with applied research: applied-ml.
• Reviewing papers for problem understanding
  • Formulation

    Classification, regression, or something else?

  • Data processing

    How was data excluded, preprocessed, and rebalanced? How were labels defined? Was a third neural class added? How were labels augmented, perhaps via hard mining?

  • Evaluation process

    How was the training and validation set created? What offline evaluation metrics did they use? How did they improve the correlation between offline and online evaluation metrics?

• How to go through each paper is discussed in the next section.

3-Pass Approach for Reading Papers

For single paper

1. Scan the abstract and conclusion to understand if the paper is useful. If it does, then skim through the headings to identify the problem statement, methods, and results.
2. In the 2nd pass, highlight the relevant sections. Helps in quickly spotting the important bits later. Take notes. For most of the papers, 2nd pass is enough.
3. Do a 3rd pass to cement the knowledge.

For multiple papers from the same domain

1. Do 1st and 2nd passes on each paper.
2. In the 3rd pass, consolidate common concepts across papers into a single note and compare the pros and cons. Doing this helps identify gaps in my knowledge. If there are gaps, then revisit the paper.

Find more here: How Reading Papers Helps You Be a More Effective Data Scientist.

Collaboration and Standard Practices

• Create shared libraries for oft-used data operations.
  • It encourages the team to contribute and thus leads to collaboration and code reviews.
  • It nudges people towards a team mindset.
• Have a single repo with training, evaluation, and inference code in one place.
  • Everybody works and reviews the same code.
  • It helps in knowledge sharing.
  • It also slows down the speed, but the pros outweigh the cons.
• Read more: Data scientists work alone and that’s bad.

I will keep updating/adding to this list as I read/experiment more.

Entering Belagavi

My first interaction in the city: ₹250 for an auto for 2 KMs. That’s just robbery! He came down to ₹100 after I told him off. Then found another auto guy who said ₹80. (This was also high, but I was tired.)

Belagavi is the (proposed) 2nd capital of Karnataka. It is a controversial city: both Maharashtra (Maha) and Karnataka want it within their borders. I don’t understand this border dispute. 😞

Update: A journalist friend, Sarayu, gave me a few pointers here. Language is the reason for this fight. Most of North Karnataka up to Goa know Hindi. Thanks to Nizam rule and the districts that border MP and Maha. And Hindi becomes more important because Bombay is closer than Bangalore. On top of that, this region becomes strategically important because it is geographically closer to upcoming districts like Karwar. There is also a spurt of educational institutions like IIT Dharward/Raichur coming up.

Language. Almost everyone understood Hindi. Everyone knew Kannada and Marathi.

Update: Belgaum is not a prosperous region and politically weak. Sarayu mentioned that since Mumbai is nearby, Belagavis go there for work. So, they learn Hindi and Marathi. This also means that youth leave for work to other cities and mostly old generations remain in the Belgaum.

Payments. After using cash everywhere in Maha, I was surprised that barring a few auto drivers UPI was widely accepted in Belgaum. It looks like Belgaum is on its way to become a smart city.

Stay. No hostels. Multiple hotels are available, but all average or below average. I chose a hotel in the market area.

Food. The first thing was Belagavi Biryani. I tried it at a famous chain called Niyaaz Restaurant (Main Branch) 📍. I’d rate it 3.5/5. A friend (met her later during the week) mentioned that Niyaaz is over-rated.

The second thing was a dessert called Belagavi Kunda. I loved it! All the sweet shops sell Kunda. You can buy 100 grams of it for ₹20 or ₹25.

Last thing was to try the Maharashtrian thali available at indie restaurants.

Tourist Points. There are two points in the Belagavi Fort area: Kamala Basadi 📍 and Safa Masjid 📍.

Story time. After wrapping up from work, I left for the Fort area. I reached my destination after walking 3 KMs and found that it was an army area. No one stopped me from going inside. I went to the mosque. Unfortunately for me, it was inside the Army protected area. My phone said 9 PM. The guards at the entry started questioning me. They let me go only when they were satisfied that I wasn’t a bad element. Of course I couldn’t see the masjid. Apparently, it is open to the general public only during Bakrid. Afterwards, while coming out of the main area where no one stopped me while entering, another army personnel stopped me. Same barrage of questions came my way. On top of that, he checked my bag and photographed my ID. It turns out that that area becomes a no-movement zone after 9 PM. Next day I visited the Kamala Basadi and ended my tourist mode. 😅

Vibe. Most of my interaction was during transactions: shop owners, hotel folks, or auto drivers. I almost never got a polite response from anyone. They were not rude either. There was no warmth in the interaction. No one smiled. Just plain dry transaction. Belagavi is the first city where I felt like that.

Story time. When I went to Kamala Basadi after the night incident, I went by an auto. I was already apprehensive of the auto guys from my 1st night. This guy turned out to be opposite. Abdul Rashid Shaikh and I talked about my reasons of coming to the town. The ₹250 story came up. He cursed all these auto guys. And the discussion (mostly me listening) went to good and bad deeds. He then took me to the Basadi and the Masjid (only to be denied entry by the army person). He then dropped me back to the Hotel. All within ₹200. I would have paid more if I had done these trips separately. This guy also suggested to skip a few cities from my itinerary. If you are in Belagavi, call for him on +91-8880866313.

I usually stay at hostels. Staying at a hotel was an experiment. Sadly, it was a failure. As a solo traveler, staying at hostels is more enjoyable than at a hotel. And usually hostels are in the cities with multiple points to explore or with good vibes. Belagavi had none. Thus, I decided to go to the cities with hostels.

Next stop: Hampi.

Abstract

• Real-time RecSys are important when customer feedback is time sensitive (eg: TikTok short-video ranking).
• The production-scale DL frameworks (PyTorch, TensorFlow) are designed with separate batch-training and model serving stages. This makes online training difficult.
• Presenting Monolith for online training:
  • Collisionless embeddings with expiry parameter and frequency filtering to reduce memory footprint
  • Online training architecture with fault-tolerance in parameter server
• Part of BytePlus Recommend.

Data in RecSys

• For many businesses driven by RecSys, better CX = real-time RecSys.
• Information from a user’s latest interaction become primary input as it’s the best signal of a user’s future interest and behavior.
• DL in RecSys
  • Wide & Deep Learning for Recommender Systems, 2016
  • Deep Neural Networks for YouTube Recommendation, 2016
  • The Architectural Implications of Facebook’s DNN-Based Personalized Recommendation, 2020
  • XDL: an industrial deep learning framework for high-dimensional sparse data, 2019
  • Kraken: Memory-Efficient Continual Learning for Large-Scale Real-Time Recommendations, 2020
  • AIBox: CTR Prediction Model Training on a Single Node, 2019
• DL in industry RecSys faces problems because of the real-world data.
• Data is different from CV/NLP tasks:
  • Features are mostly sparse, categorical, and dynamically changing.
  • Concept Drift: Training data distribution is non-stationary. (ref: A survey on concept drift adaptation, 2014)

Sparsity and Dynamism

• RecSys data has a lot of categoricals (eg: customer id, item id, item type, etc)
• Categorical features are sparse (eg: a user only buys limited items).
• Feature engineering of categorical features: map them to a high-dimensional embedding space.
• Issues with embeddings for categoricals:
  • Users and items are orders of magnitude larger than word-piece tokens in LMs. This enormous embedding table would hardly fit in memory.
  • As more users and items are added, the size would increase further.
• Current solution: Low-collision hashing to reduce the memory footprint and to allow the growing of IDs (user or item)
  • Assumptions:
    • Embedding table is distributed evenly in frequency. It is rarely true because only a small group of users or items have high frequency.
    • Collisions are harmless to model output. But it is detrimental because organic growth in embedding table size leads to more collisions.
  • Ref: Core Modeling at Instagram
  • Ref: Deep Neural Networks for YouTube Recommendation, 2016
• Thus, natural and constant demand to elastically adjust the users and items a RecSys tries to book-keep.

Concept Drift

• Underlying user distribution is non-stationary: user interests change with time (even during sessions).
• More recent data is more likely to predict change in user’s behavior.
• Mitigating concept drift: serving model should be updated as close to real-time as possible to reflect the latest user interests.

Parameter Server (PS)

• Worker machines compute the gradients.
• PS machines store parameters and updates them according to gradients.
• Two kinds: 1) training PS; and 2) serving PS. Training PS holds training parameters. Once training is complete, it is synced to Serving PS.
• Two types of parameters:
  1. Dense: weights/variables in DNN; and
  2. Sparse: embedding tables corresponding to sparse (categorical) features.
• Since both dense and sparse features are part of the TensorFlow Graph, Monolith stores them on the PS.
• The tf.Variable is for dense variables. For sparse variables, authors created HashTable operations.

Hash Table

• Representation of embeddings in TensorFlow and its limitation
  • The tf.Embedding layer uses variables to represent the dense embedding vectors. (the embedding matrix is of type Variable.)
  • The Variable construct freezes the shape of the matrix throughout the training/serving process. Thus, is it a fixed-size embedding.
  • As IDs increase with time, since the table size is fixed, ID collisions (while updating/using the dense embedding vector) would increase.
• Authors implemented a new key-value HashTable.
  • Hashing algorithm: Cuckoo hashing[Visualization + Explanation - Youtube]
  • Lookup: O(1); Insertion: amortized O(1)
  • Implemented as a TensorFlow resource operation (it likely means a TensorFlow custom layer).
  • Lookups and updates are implemented as native TF operations.
• Naive insertion: insert every new ID in the HashTable. Will deplete memory quickly.
• Insertion by frequency
  • IDs (user, item, etc) have long-tail distribution.
  • Infrequent IDs will have underfit embeddings because of less training data.
  • Model quality will not suffer from removal of these IDs.
  • Filter by a threshold of occurrences before insertion.
  • The threshold is a tunable hyperparameter for each model.
  • Also use a probabilistic filter (didn’t expand on it)
• Insertion by staleness
  • Many IDs are never visited (user inactive, out-of-date item)
  • Set a expiry time for each ID.
  • The expiry time is tunable for each embedding table: different tables will have different sensitivity to historical information.

Model Training

• Engineering steps
  1. User logs (click, like, buy) go to Kakfa.
  2. Model features are present in the another Kafka (didn’t discuss what features)
  3. Create the training example by joining the features with user logs using a Flink job.
     • First, check for the data in in-memory cache;
     • If not found, then go to on-disk key-value storage (happens in cases when user feedback arrives after days and in-memory cache is cleared to free-up the memory)
  4. Push the created training example to a 3rd Kafka queue.
  5. Push data from the 3rd queue to HDFS for offline training mode.
  6. Trigger online or offline training
  7. Push the updated parameters to the Training PS
  8. Sync the Serving PS with the Training PS
• Batch training stage
  • Ordinary TF training loop
    1. Training worker reads a mini-batch from storage.
    2. Request parameters from PS.
    3. Compute a forward and backward pass.
    4. Push the updated parameters to training PS.
  • Only train for a single pass over the data. (to mimic the online training phase?)
  • Useful when: model architecture is modified and require retraining.
• Online training stage
  • Triggered when the model is online.
  • Steps:
    1. Training worker consumes real-time data from a Kafka queue.
    2. Update the parameters in the training PS.
    3. Push the updated parameters to training PS.
• Negative sampling
  • To handle the highly skewed negative to positive sample ratio.
  • It changes the underlying distribution of the trained model: higher probability of making positive predictions.
  • Apply log-odds correction during serving to ensure the online model is an unbiased estimator of the OG distribution. (ref: Nonuniform Negative Sampling and Log Odds Correction with Rare Events Data, 2021)
• Parameter sync. between training and serving PS
  • Production models are TB in size.
  • Replacing all the parameters will take time.
  • It will also consume network bandwidth and extra storage (need to store the new parameters before replacing the old ones).
  • Solution: incremental periodic parameter sync.
    1. Sparse features (aka embedding tables): Sync the keys whose vectors updated during the last 1 minute.
    2. Dense variables (aka model weights): model weights move much slower because the momentum-based optimisers take more time to build momentum over the big data. Thus the sync frequency is 1-day. The authors found the stale weights tolerable.
• Fault tolerance: periodic model snapshots
  • Trade-off between: model quality (because of the loss of recent updates) and computation overhead (copy-pasting TB of data)
  • Snapshot frequency: 1-day. Experiments revealed that performance degradation was tolerable.

Evaluation

(I will skip the experiment setup and jump over to the results.)

The Effect of Embedding Collision

• Model with collisionless embedding vectors consistently outperform the one with collision.
• Independent of training epochs and concept drift (non-stationary training data)

Online Training vs Batch Training

• Online training has better performance than the batch training.
  • AUC of online training models: evaluated by the following shard of data.
  • AUC of batch training models: evaluated by each shard of data (?)
  • General AUC delta ranged between 0.20 (5hr interval) to 0.40 (30 min interval).
• Smaller parameter sync interval (or higher parameter sync freq.) performs better than the larger intervals.
• Based on these results, best sync frequency for sparse features that the systems could endure was 1 minute.
  • Assuming 100,000 IDs with 1024 vector size are updated each minute: ~400 MB (4 KB * 100,000) network transfer per minute.
• Sync frequency for dense features is 1-day (every midnight) as they update slowly.

Is the 2nd figure correct? The 5hr sync interval model should degrade till the sync happens. After sync, it should have similar AUC as other models. It should then degrade again from that point until the next sync. That is not happening here. What am I missing?

PS Reliability

• Hypothesis: minute-level parameter syncing should mean frequent snapshots.
• Wrong. Observed no loss in the model quality even with 1-day snapshot interval.

• Below excerpt explains the reason:

  

• Lesson: don’t take frequent snapshots and save resources.

Summary Conclusion

• The paper proposed the following:
  1. Cuckoo HashMap based collisionless embedding tables
  2. Online training and parameter sync architecture
• With extensive experimentations (both offline and online) they showed that:
  • Collisionsless embedding table has a positive impact on the model quality (AUC gains ranged from 0.20% to 0.40%)
  • Online training performs better than batch training in RecSys setting.
  • Higher parameter sync freq is better (1 minute for prod systems); and
  • It is okay to have a smaller parameter snapshotting frequency (1-day for prod systems)
• This paper was a write-up of engineering tricks that ByteDance employed to build their RecSys.
• The few nuggets of ML that I noticed:
  • Apply log-odds correction to the data in online serving to make up for negative sampling.
  • Online real-time model at ByteDance is a multi-tower architecture where each tower is responsible for learning a special kind of user behavior. (Is than an allusion to multi-objective ranking through different towers?)

Overall, this paper adds to my belief that a successful system requires clever engineering.

One does not get mentoring lessons at school. There is not enough time to read books on effective coaching. My only guides have been two specific rules.

First is thinking about how I would have wanted my mentor to coach me. And then coach my mentee the same way.

The second is to observe my current mentors. Notice the techniques that enable my growth. Inculcate these methods in my mental model. Similarly, spot where they are ineffective and learn to avoid them.

The Flaw

Recently, my boss inadvertently showed me a flaw in my first rule.

Good design is thorough down to the last detail. Nothing must be arbitrary or left to chance. Care and accuracy in the design process show respect towards the consumer.

- Deiter Rams’ 8th principle of Good Design.

That embodies my personality. I believe that my work should be proper (read, perfect). And, at times, I become rigid about maintaining that standard. Thanks to this, my output is usually of good quality (not bragging 😅). That satisfies me. The gratification keeps me intrinsically motivated. Thus, I continue to work like this.

My rule assumes how I would have wanted my mentor to coach me. That means that I presume my coach to have equally high standards. It is a flaw. Since everyone is different, my way of operating does not work for everyone.

Patience is a Virtue

I also have a patience problem. When I think I can do something faster and the other person takes more time, then that annoys me. I have worked on this quite a lot in the last few years. But there is room for improvement.

What is Next?

My overarching goal is to be a good leader. I believe a good leader is also an effective mentor and coach. So, I am actively going to make myself a good mentor.

The following are the next steps for me:

1. Stop judging by the yardstick of “is this how I would have done it?”
2. It is okay if things are not how I thought they would be. If it is 80% there, it is good enough.
3. If there are areas of refinement, then definitely point them out.
4. Stop thinking that I could have done it quickly. Get comfortable with others being slow/fast.

Let’s see how it goes.

Abstract

• A hybrid model with two parts:
  1. Embedding generation: generate implicit embeddings of properties.
  2. Deep prediction and ranking model.
• The model performed well over the existing collab-filtering model.

Situation/Context

• OYO’s current recommendation system
  • Graph-based Collaborative filtering model
  • Optimised on browsing data as user feedback
  • Objective: CTR
• DL provides an opportunity to improve the system.

Lit Review

• Conventional RecSys algos:
  1. Collab-filtering,
  2. Content-based, and
  3. Hybrid
• YouTube’s 2016 paper has demonstrated that DL-based RecSys can give SOTA results on high-volume data.
• MF only considers the linear combination of user and item latent vectors. Whereas, DL can capture non-linear user-item relationships.
• DL reduces the feature engineering efforts.
• RNN facilitate temporal behaviour of user-item interactions: useful for session-based sequential recommendations. Conventional algos don’t capture this.
• Research on modelling user behaviour sequences using LSTM or GRUs
  • Sequential User-based Recurrent Neural Network Recommendations, 2017
  • Recurrent Neural Networks with Top-k Gains for Session-based Recommendations, 2018
  • Session-based Recommendations with Recurrent Neural Networks, 2016
  • Neural Attentive Session-based Recommendation, 2017
  • Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks, 2017
  • Recurrent Recommender Networks, 2017
  • A Dynamic Recurrent Model for Next Basket Recommendation, 2016
• This Airbnb paper (summary) takes the sequence of listing ids clicked by the users and trains a skip-gram word2vec model on it. And then rank using these embeddings.
• The authors of this paper mention that they improve it by adding entity features along with click data.

Methodology

• Embedding generation: generates embeddings of the hotels (intermediate output of the next step).
• Prediction and ranking model: gets top-n recommendations-based on the following inputs:
  • The sequence of browsed hotels
  • Embeddings of the browsed hotels
  • Rating tokens of the browsed hotels
  • Realisation tokens of the browsed hotels

• What was the candidate list of hotels? High-rated hotels?

Embedding Gen

• Explicit feedback requires effort from the customers; hence, ratings are sparse.
• Browsing data as user’s implicit feedback; thus, no sparsity.
• In this work, implicit features were derived using an RNN.
  • Embeddings were the intermediate output of the model training process.

Prediction and Ranking Model

• Objective: realised bookings (conversion along with the realization of bookings)
• Implemented the following four methods: RNN, GRU, LSTM, and BiLSTM.
• Training data:
  • 1 million users
  • Sequences of their clicked hotels within a session
• Pre-processing: padded and limited to 15 hotels.
• Model objective: the probability of the user for realised booking at high-rated hotels.
• Proposed architecture (disclaimer: I couldn’t grok it from the paper)

  1. Embedding layer: 100 dim

      torch.nn.Embedding(num_embeddings=all_hotels, embedding_dim=100)
     

  2. Embedding concat layer (torch.cat())
     • Not sure why they concatenated the embeddings.. The embedding tensor should have been input to the RNN layer. Otherwise, no recurrence will happen.

  3. 2 BiLSTM layers

      torch.nn.LSTM(
          input_size=1530,
          hidden_size=512,
          num_layers=2,
          bidirectional=True)
      )
     

  4. Flatten layer (torch.nn.Flatten())

  5. 4 ReLU dense layers

      l1 = torch.nn.ReLU(torch.nn.Linear(512*2, 512))
      l2 = torch.nn.ReLU(torch.nn.Linear(512, 256))
      l3 = torch.nn.ReLU(torch.nn.Linear(256, 128))
      l4 = torch.nn.ReLU(torch.nn.Linear(128, 1))
     

  6. Softmax layer
  7. Output layer

  

Embedding Evaluation

• Embedding dimension: 100
• Get the top 10 similar hotels for all the hotels in the training dataset using cosine similarity.
• Four accuracy metrics:
  1. Location
  2. Distance
  3. Price
  4. Ratings

• Metric formulation

  \[\text{Sim Index @ x} = \frac{\sum_{i=1}^{H} \text{sim@x}(\text{top-10 hotels}, i)}{H} \\\]
  • \(x\) can be any of the following: Location, Distance, Price, Ratings
  • \(H\) is a set of all query hotels;
  • \(\text{sim@x}(\text{top-10 hotels}, i)\) is the similarity score for metric \(x\).
  • Ranges between 0 and 10.
• \(\text{sim@Location}\): fraction of top-10 hotels lying in the same city as the query hotel \(i\).
• \(\text{sim@Distance}\): fraction of top-10 hotels that are within a 20km radius of the query hotel \(i\).
• \(\text{sim@Price}\): fraction of top-10 hotels that are within +/-15% of the price of the query hotel \(i\).
• \(\text{sim@Ratings}\): fraction of top-10 hotels that are +/-1 rating from the query hotel \(i\).
• Following are the evaluation results with the winner highlighted:

  

• Qualitative eval also yielded positive results.

Ranking Model Evaluation

• Offline evaluation metric: Hit Ratio, MRR

• \(\text{Precision@k}\) or Hit Ratio: fraction of users for which the booked hotel was among the top-k recommendations.

  \[\text{Precision@k} = \frac{U_{hit}^k}{U_{all}}\]
• 15 total model variants:
  • 3 variants with basic RNN
  • 4 variants each with LSTM, GRU, and BiLSTM
• Selected one variant from each model type-based on validation results.
• Created a dataset aligned with the real-time environment. (Session logs?)
• Out-of-time validation on this dataset.
• The BiLSTM variant was the best-performing model.

  

• Online evaluation metrics:
  • Realized bookings at high-rated hotels.
  • C*R (multiplication of booking conversion and realization of bookings) at high-rated hotels.
• Observed lifts of 3% to 6% in realized hotel bookings across different geographies.

Review Conclusion

• The paper proposed building a DL model with two parts: embedding gen and ranking model.
• The embeddings are the intermediate output of the ranking model. Not sure why it is called a separate model in the paper.
• The model is an important part of this paper, yet
  • It does not discuss the training data construction in detail.
  • Few left-out details about the architecture made it difficult to comprehend it.
  • There was no discussion about inferencing and the candidate set of restaurants to rank.
• The embedding evaluation framework was comprehensive and quantified the effectiveness of the embeddings.
• Model evaluation methodology followed the standard process of train-time validation and out-of-time validation steps.
• One thing lacking was comparison with tree-based models like gradient boosted trees which have shown good performance in recommendation tasks in both industry and research.

You can follow the story of what I wanted to do and how I did it. Or jump to the solution.

Situation

• The PySpark MLlib (DataFrame-based, RDD-based) does not support kNN algorithm.
• The multiple SO questions (1, 2, 3) did not help.
• I did not get a chance to try the available open-source code samples (1, 2).
• There were two stable-ish solutions available:
  1. Use Spotify’s library called Annoy.
  2. Use Scikit-learn’s implementation of kNN.
• Both methods only use a single node. So, the benefit of Spark’s distributed processing goes out of the window.
• That’s a deal breaker because I have a large data set (~20+ mill records) with long vectors.
• I picked Annoy because I found it first. I discuss at the end why Scikit could be more performant.

The task is to parallelise the Annoy code across multiple nodes of the Spark cluster.

Solution

• A hint about the solution is present in this SO Answer.
• For both Annoy and Scikit, the approach is as follows:
  1. Build the index or fit the model on a single node. Nothing is distributed here.
  2. Broadcast the index (or model) across the cluster to find the nearest neighbours of a given vector.

Building the index

• I first tried to use spark-annoy. It is in Scala. The benefit of this library was that we could build the index in a distributed manner. Unfortunately, I could not figure it out.
• The default was to use the iterative approach of building the index on a single node.
• The following is the method to build the index:

1
2
3
4
5
6
7
8
9
import pandas as pd
from annoy import AnnoyIndex

def build_annoy_index(vectors: pd.Series, dim: int, num_trees: int = 100):
    t = AnnoyIndex(dim, metric='angular')
    for index, vector in vectors.items():
        t.add_item(index, vector)
    t.build(num_trees)
    return t

• The for loop makes it a long-running process if the data is large. Sadly, it is unavoidable.
• Note that the type of vectors is pd.Series. I used pandas to get the goodness of indexes. It can be a list or any other iterable. It should be an iterable irrespective of its type. That means either of the following:
  1. Run .collect() on the Spark DataFrame;
  2. Turn the spark DataFrame into a pandas DataFrame.
• That will bring all the data to a single node. So, it can potentially lead to OOM error.

Finding Nearest Neighbours

• We can parallelise this step.
• We have to broadcast the Annoy index across all the nodes of the Spark cluster.

• The Annoy indexes are memory mapped.

  It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.

• It will fail if we broadcast it using sc.broadcast(t). This SO answer discusses this issue.
• The solution: write the index to a file and send the file to all the workers to load.
• Use sc.addFile() to send the file to the workers.
• Use SparkFiles.get() to get the file path and load it in the worker node.
• Here is the method to load the Annoy index in the worker nodes:

1
2
3
4
5
6
def load_annoy_index(index_file: str, dim: int):
    from annoy import AnnoyIndex

    index = AnnoyIndex(dim, metric='angular')
    index.load(SparkFiles.get(index_file))
    return index

• I call the below method to get the nearest neighbours of a set of index ids:

1
2
3
4
5
6
7
8
9
def find_neighbours(index_file, top_n, dim, item_batch):
    index = load_annoy_index(index_file, dim)

    # get similar items
    sim_items = []
    for item in item_batch:
        top_n_items = index.get_nns_by_item(i=item, n=top_n)
        sim_items.append((item, list(enumerate(top_n_items))))
    return sim_items

• The item_batch is the list of Annoy index ids.
• The function returns the list of (annoy_item_index, [(rank_1, sim_item_1), ..., (rank_n, sim_item_n)]).
• For example, here is one such item from the list:

(248, [[0, 248], [9, 284764], [3, 86148], [6, 265812], [7, 508155], [2, 48388], [10, 58786], [1, 154653], [5, 364419], [4, 4444], [8, 89955]])

• To validate the function, the most similar item corresponding to the query item should be itself. In the above example, the query index 248 ranks 0 in the top similar items.
• To get the nearest neighbours by vectors, you pass the vectors in the item_batch and use the get_nns_by_vector method.
• I keep my find_neighbours method generic for the following parameters:
  • Annoy index file name: I can have any name based on my use case.
  • Top n items: number of top items I want to retrieve.
  • Dim: The dimension of the vector can vary depending on the various ML techniques (LDA, DL, etc.)
• We have only written the method to find the nearest neighbours. How do we call it in a distributed manner? This SO answer answers that too.
• The answer is mapPartitions. This method will apply the passed function to each RDD partition. Go through the answers of this SO question to know more in detail.
• I will pass find_neighbours to the mapPartitions, and it will return an RDD with the nearest neighbours list.
• But my find_neighbours implementation takes four parameters, and there is no way of sending **args inside the mapPartitions.

• I use the inbuilt python partial() function from the functools module.

  The partial() is used for partial function application which “freezes” some portion of a function’s arguments and/or keywords resulting in a new object with a simplified signature.

• Here is how my final function looks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def build_index_get_similar_items(vector_df, index_file, top_n):
    vectors = vector_df.vector
    sparkvector_ids = sc.parallelize(vector_df.index.values)

    # build and save index
    index_file = build_save_annoy_index(vectors, dim, index_file, BASE_DIR)
    print(index_file)

    # add index file to the driver files
    sc.addFile(index_file)

    # get similar items
    find_neighbours_ = partial(find_neighbours, index_file, top_n, dim)
    similar_items = sparkvector_ids.mapPartitions(find_neighbours_).collect()
    return similar_items

• The build_save_annoy_index() method builds the index, saves it to a file, and returns the file path.
• Finally, we see the use of sc.addFile(index_file).
• The find_neighbours_() is the partial function. We froze the index_file, top_n, and dim. This function now only expects a single RDD as input. And this is what we wanted for the mapPartitions() method.

Saving Results

• I take the similar_items list and convert it into a pandas DataFrame.
• Map ALL the Annoy index ids with the actual item ids. That includes all the index ids of the top-n similar items list.
• Convert the pandas DataFrame to a PySpark DataFrame.
• Save the PySpark DataFrame into a delta table.

Result

• I was able to parallelise the kNN search based on Annoy using mapPartitions.
• On ~500k records, the run time was down from 8 minutes to 2 minutes.
• On ~10 million records (with an index built from ~500k records), the run time was ~1 hour.
• On ~10 million records (with an index built from ~10 million records), I got an OOM error. 🥲

What’s Next

• Find the reason it is going OOM.
• Converting the pandas DataFrame to PySpark DataFrame is expensive. I want to explore if I can directly go from pandas DataFrame to the delta. Ref: 1, 2.
• Since the Scikit has vectorised training and inferencing, its kNN would likely be faster. This post shows how to do it. I would probably replace the map with mapPartitions.

⌛ 10:00 PM

• Started for the bus boarding point.
• Bus was 15 minutes early for the first time ever. Asked the cab driver to speed it up.

⌛ 10:30 PM

• Boarded the bus. It started within a minute.
• After getting on the bus, gave a status update to mum 🙋‍♀️, A friend also called later.
• Slept at ⌚ 12:20 AM.

📆 29th April

⌛ 06:00 AM

• Woke up.
• Watched the sunrise from the bus. The original plan was to go to the Rock Beach 📍 from Pondicherry 📍 bus stand and watch the sun rise.
• Pinged a few friends about my arrival.
• I got worried when the driver took a different route. He later converged on the correct route.

⌛ 07:30 AM

• State bus for Auroville beach 📍. Rs. 12 ticket. Empty seat.
• Feeling the heat a bit.

⌛ 08:00 AM

• Walked to Vagabond 📍 from bus stand.
• Sriram 🙋‍♂️, the property manager, showed me to the top bunk bed on the ground floor.
• Met Saiprasanth 🙋‍♂️ and Santosh 🙋‍♂️.
• Brushed my teeth and went to the terrace.
• Met two new people from Bengal/Odisha. They both have a recruiting company in Bangalore. They came 30 minutes before me. They had booked the Vagabond Airbnb.
• Felt hungry. Sriram 🙋‍♂️ suggested Auroville Bakery 📍.

⌛ 08:20 AM

• Took one of the property bicycles to the bakery. Cycled ~2 KMs.
• Did not want to experiment. So ordered Idli and Red Rice Cheese Dosa. The dosa was not good. :/
• Breakfast was filling.
• Came back feeling super hot.

⌛ 09:20 AM

• Met Tanusha 🙋‍♀️ and Anushka 🙋‍♀️.
• Explored the property’s garden and terrace.
• So many Jackfruit trees. A few mango trees as well.
• Worked on the Swiggy blog for a few minutes.

📆 3rd May

⌛ 10:30 PM

• Came to the terrace.
• Migrated the blog google analytics tag.
• Started Pondi travelogue. Published it.

📆 4th May

⌛ 12:30 AM

• Went down to to sleep.
• Read Name, Place, Animal, Thing 📖 for 30 minutes.

⌛ 09:00 AM

• Finished Anki.
• Breakfast: mango shake and 1 chila. Naveen 🙋‍♂️ prepared it for everyone at the hostel. Amazing guy. He loves to do this. This is a very different feeling for me. A good different.

⌛ 10:00 AM

• Started work. Working on a terrace is a mind blowing experience.

⌛ 01:30 PM

• Went to Sustenance Cafe with Naveen 🙋‍♂️ on Santosh 🙋‍♂️’s scooty.
• I had a South Indian meal (it was good) for 150 bucks. Naveen 🙋‍♂️ had 3 chapati and sabji for 90 bucks (not so good).
• I met a French guy named Benoit 🙋‍♂️. He is on a two-week long tour in India. He’s going to go to Mysore tomorrow. From India, he is going to London. He is a technology writer. I told him about Swiggy.
• We shared Instagram deets.
• Inquired about a scooty for us. Rs 3000 for a Pep for a month. Rs 3500 for an Access for a month. Naveen 🙋‍♂️ and I will share it till he is here. Post that it will be with me.
• We will get it in the evening.

⌛ 02:30 PM

• Back to working on the terrace.
• In a meeting, my boss asked me how my workation was going. Told him all the fun things I did.

⌛ 06:00 PM

• Fellow hostelers left for a swim on the Serenity Beach 📍. I was in a meeting at that time. But they were thoughtful enough to leave the key to a scooty for me.
• I drove it to the beach without any incident. It was the first time I had ridden a scooty on the main road. Traffic was light so it was easy. I am feeling more confident now. I forgot the road leading to the beach, but a few shopkeepers helped.
• Joined these folks. It was a low tide but waves were big enough for light surfing. Sriram 🙋‍♂️ was on his surf board. Santosh 🙋‍♂️, Naveen 🙋‍♂️, and I swam in the water.
• I had a goal of going for a swim three times this week. I achieved that.

⌛ 08:36 PM

• Went to Jeffi Restaurant 📍 for dinner. With Naveen 🙋‍♂️, Santosh 🙋‍♂️, and Ved 🙋‍♂️. This place was suggested by Sriram 🙋‍♂️.
• Finished the objective of visiting three new restaurants.
• Today is Ved 🙋‍♂️’s last night in Pondicherry 📍.
• We had tandoori chicken, chettined chicken, phulka, and butter roti. Spent Rs 915.
• Went to Richy Rich Ice Creams 📍. Ate a 120 bucks scoop of Belgian Almond Chocolate ice cream.

⌛ 10:00 PM

• Reached the Vagabond 📍.
• Came to terrace and relaxed till ⌚ 2 AM. Slept by ⌚ 2:20 AM.

📆 5th May

⌛ 09:15 AM

• Woke up. Finished Anki.
• Naveen 🙋‍♀️ had texted saying that two chocolate croissants 🥐 where kept on the dining table. Again, so thoughtful of him.

⌛ 10:25 AM

• Finished my croissants 🥐. Not warm, but still tasted good.
• MacBook started updating. Useless laptop. Finally, finished updating right before I had to take an interview. Phew.

⌛ 01:10 PM

• I wanted to try out Sree Andhra Tiffins 📍. Others didn’t want to go out. So, went alone.
• Went by scooty. I was able to maneuver through the traffic, red lights, and local Pondicherry market. Felt accomplished.
• Plan of gaining confidence in riding a 2-wheeler is going well.

⌛ 01:35 PM

• Ate a classic meal. It was a large and sumptuous meal 😋.
• It was served on a banana leaf. It contained: three dishes, rice + podi + ghee, pickled garlic, rice + sambhar, rice + rasam, rice + curd curry, papadam, and mango slices.
• All this for 110 bucks.
• Got a meal packed for others.
• The person sitting opposite to me has been eating at this place since 2008. Whenever he visits to Pondicherry 📍, he always eats his meals here.

⌛ 02:02 PM

• I had to come back to the hostel by ⌚ 2:00 PM for a meeting. Unfortunately, I was still at the restaurant.
• Joined the call from phone listening everything on my earphones. Hard to manage because I had to stop at places to participate.
• Reached back at ⌚ 2:30 PM

⌛ 05:30 PM

• Met Dinesh 🙋‍♂️. He checked-in today. He drove from Bangalore 📍 to Pondicherry 📍. He is an iOS developer.

⌛ 06:20 PM

• Reached the Serenity Beach 📍 with Naveen 🙋‍♂️, Santosh 🙋‍♂️, Sriram 🙋‍♂️, and Dinesh 🙋‍♂️.
• I drove the scooty with Naveen 🙋‍♂️ as a pillion rider. 💪
• I swam three long laps. I caught two waves 🏄‍♂️ with Sriram 🙋‍♂️’s help.

⌛ 08:20 PM

• Naveen 🙋‍♂️ had started cooking 🧑‍🍳.
• He was preparing okra dish.
• I assisted him. I cut onions. I prepared chapati dough. I also prepared banana shake. Both of us prepared 9 chapatis.
• It took us ⌚ 1.5 hours to prepare the whole meal 🧑‍🍳. We talked about multiple things while cooking.
• Five of us ate together. It was nice.

⌛ 10:00 PM

• Told my mum 🙋‍♀️ about today.
• Talked to Naveen 🙋‍♂️ about his CFA certification.
• Wrote today’s tavelogue.

📆 6th May

⌛ 12:00 AM

• Listening to Santosh 🙋‍♂️ and Dinesh 🙋‍♂️ talking. They are drunk.
• Dinesh 🙋‍♂️ is going to resign on Monday and move to Canada in August.
• Dinesh 🙋‍♂️ went to get more beers. Santosh 🙋‍♂️ started talking about Vagabond 📍. He says, “Every week I think about moving to other city or place, but then I don’t.” He goes on: “It is never the place. It is always the people. People make it worthwhile. I always meet someone new.”
• He came to Pondicherry for only 2 days. It is his third month.
• One of the friends I knew before I came to Bangalore told me that if you don’t have an agenda of why you want to come here then this place is going to grow on you. I wonder how it will be.

⌛ 02:10 AM

• Time to crash, but went down and read Name, Place, Animal, Thing 📖.

⌛ 09:00 AM

• Woke up, and did the usual. Work started early due to a few P0 issues.
• Market low. “Buy on dip.”

⌛ 01:40 PM

• Lunch @ Baghiraa Cafe & Co Work 📍 with Naveen 🙋‍♂️.
• Bought a few personal care items (shower gel, and moisturizer) from Farm Fresh 📍.
• Came back to the hostel. Of course, I was using scooty for all the transportation.

⌛ 06:00 PM

• Biked 🚲 to Serenity Beach 📍. Finished four laps today. 💪
• Came back on scooty with Santosh 🙋‍♂️ while Naveen 🙋‍♂️ cycled back.

⌛ 08:45 PM

• Ragavan 🙋‍♂️ came to pick me up. (context: Ragavan is a dear friend from school. He was in Pondi today, so we planned to catch-up.)
• We went to Cafe 73 📍 on his cousin’s bullet. Coincidence was that Cafe 73 is a bike themed cafe. There is a bike on display when you enter. All the walls have some bike art. Sriram 🙋‍♂️ had suggested to try out non-veg burgers 🍔 from here. Ragavan ordered The Ferrari F40 and I ordered The Turbo Charger. And burgers were succulent.
• We started with work and life. We discussed Pondi and nearby places. We discussed my city hopping plan. We ended with him inviting me to his place in Chennai on a weekend.
• We went to the Serenity Beach 📍 to relax. Sat on the rocks. Listened to the waves crashing against other rocks. Talked about random things.
• It took me back to 2018. Ragavan and I were in Mysore 📍. We were meeting after 6-7 years. We went to Mysore to attend the Mysore Dasara celebrations. It ends with the whole palace lighted with 100,000 light bulbs. After it ended, the workers started packing up all the chairs and tents. We were sitting on a pair of chairs in the last row. And we started going through the 7 years worth of events from each other’s life. Both of us, vicariously experienced each other’s life. It was an amazing end to our days.
• He dropped me back to Vagabond 📍.

⌛ 11:00 PM

• Came on terrace to write this log. Santosh 🙋‍♂️, Dinesh 🙋‍♂️, and Tanusha 🙋🏻‍♀️ were already here.
• Talked to a friend about product management and design.
• Six college students joined us. They were staying at the Airbnb.
• Suddenly, Gales started. It rained a bit. And then there was a power cut. The climate became cool and refreshing. No humidity in the air.
• Asked Santosh 🙋‍♂️ what else defines him besides being a fitness coach. He took time before answering. His answer was: inventions, start ups, and product design. He had a course about product design. He wants to build a product in fitness domain.
• He is traveling since November 2021. He had to come back from Dubai due to Covid related incidents in family. After settling everything, he couldn’t go back to Dubai. He started traveling: Rishikesh 📍, Mysore 📍 and now, here.

📆 7th May

⌛ 02:45 PM

• Started working on this log.
• Off to sleep.

⌛ 12:30 PM

• Just woke up. Those six college students left.
• What are you going to do after your CFA exam on 24th? Naveen answered that he is going to stay home and earn money till the results come. Mode of earning money: trading. He was confident about it. If he had the freedom, he would have been traveling all around to play all kinds of sports.
• He volunteered at Mantra Surf Club 📍 earlier this year. Coincidently, my beginner surfing course was also at Mantra. We talked about the instructors and some surfing tricks. We may go for surfing next week. ‘tis the surfing season in Pondi.
• He has also done the beginner and intermediate Skiing courses from JIM & WS 📍. I have been trying to plan for the same course since last year. Hopefully, next year. 🤞🏼

⌛ 01:45 PM

• Lunch @ Neem Tree Cafe 📍 with Naveen. I had Red Rice and Dosa meal and he had chapati sabzi meal. It was disappointing. Both the meals had the same dishes. That meant I had to eat my dosa with lauki (gourd) instead of sambhar or some chutney. This restaurant was inside Auroville 📍. Not sure what to expect from the other Auroville restaurants in my list.
• Came back and slept again. Today was a slow day.

⌛ 06:00 PM

• Biked 🚲 to Serenity Beach 📍. Finished four laps again. 💪

⌛ 08:15 PM

• Dinner @ Baghiraa Cafe & Co Work 📍 with Naveen and Santosh.
• We discussed workout and nutrition with Santosh. I wanted to know how to tighten the skin around my abs. Answer: Planks and Wall Press Dead Bug. We talked about eating sugars. He also mentioned the importance of fat in the body. Apparently, leanest body will require 6-7% fat in the body. Normal is ~20%. Looking at my physique, he said I would be under 20%. Anything above 15% and under 25% is good.
• I gotta take a Nutrition 101 course.

A church! (Cross gesture) Jesus, pls help me find a good flat. Ooh, ooh. A temple. (Praying hands) God, pls help me find a good flat within my budget. Wow! A mosque too! (Praying hands) ya Allah, pls help na.

That was my first week in Bangalore.

Fellow TMs and guests. Have you ever longed for the past? Have you ever wished to experience a memory again? Have you ever felt nostalgic?

Nostalgia is a curious feeling. Most of the memories are trivial. But, they always bring a deluge of thoughts with them. Today, I am going to talk about three specific triggers from my life. First, music and how it makes me feel proud of myself. Second, the summer season and its relaxing effect on me. Lastly, my first week in Bangalore and how it motivated me to start a new thing.

So, come, let’s take a trip down my memory lane. (smile)

Music.

(Singing like a metal head) “Whiskey in the Jar-o” is a song by Metallica. I first heard that song at 2 am on the radio. I was “studying” for my board exams. Metallica reminds me of how I became a night owl. I have pulled off numerous all-nighters by now. For me, it is the best time of the day.

A friend introduced me to Coldplay. This band reminds me of my crush and how I was too chicken to pursue her. I am more daring now. And, I wish I could go back.

Finally, the Explosions in the Sky. This instrumental band accompanied me on all my late-night walks. Walking did not help me shed my chubbiness. But, it did have a therapeutic effect on me. To this day, whenever I want to clear my mind, I go on a late-night walk while listening to the Explosions in the Sky.

Songs of these bands always remind me of who I was. And where I am now. I am sure that the ambitious younger me would be elated to see the current “me,” equally ambitious. And that always makes me feel proud of myself.

Summer.

Since first April, the heat has picked up in Delhi, and my mum has been relentlessly cribbing about it. She says, irritated, “It is too hot!. Summer is early this year. Why does it even exist?” I, on the other hand, enjoy Summer. So we go into this banter of which season is best. Of course, I don’t need nostalgia to remind me of this banter. My mum does that every April.

But, summer reminds me of very distinct periods of my life.

Imagine this: The Sun is hot. You feel needles pricking you all over. You take your clothes off and jump in the cold, clean water of Yamuna. After a long dip, you go to a nearby field, pluck a big watermelon and gobble it up. Summer reminds me of that 7-year-old me.

School days. How many people here remember the games period? Huh? How about school buses? I used to walk home from school. Hopping from the shadow of one tree to another, only stepping into the sun when there was no shadow remaining. Upon reaching home, I used to build cool stuff from the broken things at home. Summer reminds me of those carefree school days.

Summer also reminds me of the 1st and 2nd waves of Covid. I stayed in Bangalore. And so did my flatmates. We cooked together. My chapati making skill is top-notch now. We played inside our home. We worked out together. One flatmate and I shaved our heads together. And the cherry on top was how we ended the summer with a carefully planned road trip to the Western Ghats.

Summers are truly wonderful.

My first week in Bangalore.

It happened on the 3rd or 4th day. I was flat hunting and decided to explore Bangalore on foot. Seeing those three holy places in the vicinity of each other made me realize how incredible India is. And how, it turned out, it was an equally incredible experience for me.

I am about to start an experiment in the coming weeks. It is going to require a titanic change in my lifestyle. Frankly, I am scared to even think about it. Thankfully, this fear helped me. It reminded me of my first week in Bangalore. I remembered how much I enjoyed being on my own. For the first time, I was not under my parent’s care. I was making all the decisions. Instead of being afraid, I was super excited! Like the cold seeps through a cracked door, excitement seeped into me. That unexpected nostalgic moment motivated me to go ahead and take that step.

It is the smallest thing, eating a watermelon at a riverbank. But it is shocking how I want to do it again and again today. Likewise, I could have never imagined that my chaotic first week in Bangalore would motivate me in future.

We seldom get to experience nostalgia in our busy lives now. Hence, here is something I want you to do after this meeting. Go reminisce about the past. Past that fills you with warm and fuzzy feelings. And if you feel you do not have such a memory, I implore you to make them. It does not matter how significant or insignificant they are.

Because you never know which tiny event would make you smile, help you discover yourself, or later motivate you to take the next step.

The second project was to deliver a speech having a purpose. The Icebreaker project (1st speech) gets you started on the path of public speaking. The second project makes you understand the importance of purpose in any communication.

The upcoming section contains my final speech. I expand on the purpose and its importance after that. Then I discuss my takeaways from this project, followed by the conclusion.

Explore or Exploit

Can I see a show of hands if anyone here orders food from Swiggy or Zomato? (wait) Thank Now let me see a show of hands from those who ALWAYS order from the same restaurant. (wait) [No one. That is obvious. Right? You will also try out new restaurants.] This is the last one. Let me see a show of hands from those who ALWAYS try out a new restaurant. (wait) (smile) [That was another crazy question.]

Most of us are the same: we either want to order from our safe zone or try from uncharted territory. Listening to it in this context may sound obvious. After all, you should have both choices available to you. How else are you going to find a new place?

Unfortunately, we often miss this in the context of our life. Maintaining a balance between exploration and exploitation is the message I want to convey today.

I started exploring cycling when I was 9 or 10 years old. I did not plan on being the third wheel between my cousin and his bike. So, I learnt it in a few days and switched into the exploit mode of riding it daily.

After gaining confidence, I decided to ride my elder brother’s bike. The next day, in the early morning, I pondered about taking the risk. Why would it be risky? Because when on the bike, my toes never reached the floor.

You know the urge to do something that people older than you are doing can make a child do anything. I decided to take the risk.

I could maintain the balance. As I could only pedal in half circles, momentum was building up slowly. Now, of course, something bad happened. I suddenly hit one of those nasty speed breakers. The likes of which, along with your speed, also break you. After the impact, I found myself off the saddle and hanging on the frame. My feet were nowhere near the pedals and scrambling for support from the road. Since my toes could barely reach the road, I knew if I hit the brakes now, I would fall and hurt myself. At some distance, I saw a bunch of hay on the footpath. I manoeuvred the bike to break my fall on that cushion. There were a few laughs from the bystanders. I also laughed with them.

In retrospect, it was a tiny incident, but for a 10-year-old kid, it was a confidence booster. I remember feeling proud of myself for coming out of it without any injury. And I was glad I decided to explore riding on my brother’s bike.

The explore-exploit equation gets warped at school.

You are required to study a fixed set of subjects. Under the influence of that mandate, I never thought to explore any of my subjects. That thinking changed when a tutor got me into the habit of reading my textbooks. After that, subjects were not something that I had to study. They became different avenues for me to explore.

My favourite subject was physics. I was obsessed with Einstein and Feynman. I loved reading about the theory of relativity. Schrödinger’s cat made atoms and waves exciting. Concepts of thermodynamics made life practical. I even envisioned the future Shivam building a successful nuclear fusion reactor to satisfy all our clean electricity needs.

I was also caught by the outer space bug. Here is a trivia that blew my mind: you witness the past when you see the stars during the night. That is because they are so far that it took light years to reach you.

Then I got to explore, not study, but explore computer science in 11th grade. You could code anything up if you had the creativity and a logical mind. I was immediately hooked. I romanticize those years as a time when I had a love triangle with Physics and Computer Science.

Of course, every good thing comes to an end. When it was time for college, I defaulted to exploitation. The reality, seen through the eyes of peers, teachers, and parents, persuaded me to opt for Computer Engineering.

The thing is, I have never regretted this decision. There were numerous subfields to explore within computer science. My career shaped into what it is today because of those explorations.

I started my Computer Science engineering in the year 2012. Data Science and Artificial Intelligence started booming. Data Science suggests that we can solve problems in any domain when simple maths is applied to data. That was fascinating to me. It also had many concepts of Physics, my first love, mixed in with it.

I explored many interesting subfields in computer science before settling for Data Science. When the question came to decide on a career, I had already covered a lot of ground in Data Science. And its interdisciplinary nature compelled me to exploit it further.

I give full credit to my explorations for Data Science as a career choice. It has given me exposure to different industries and how the world works. And deciding which industry to try next is also always driven by my drive to explore.

I am not talking about Covid here, but I feel we are undergoing a pandemic where people lack the approach to find what they can exploit. And that is because they do not explore enough.

I know many people who have taken this framework to the extremes and are incredibly successful and satisfied.

So my message for you today is to explore as much as you can. The moment something enthrals you, go into exploitation mode. But never stop exploring. Exploration is what makes you, you.

Purpose and its Importance

Books have a purpose; this blog post has a purpose; like every communication, a speech also has a purpose. We are instinctively aware of this whenever we say something.

Nevertheless, I was perplexed when I started writing this speech. I had a vague idea of what I wanted to talk about, but I could not articulate it. Response from my mentor also did not help me. The project resources provided by Toastmasters succoured me.

According to the guide, every speech has a generic purpose: inform, persuade, entertain, and inspire. Every generic purpose will also have a specific purpose. It is one sentence that summarizes the objective of your speech. The general purpose of my speech was to inspire. The specific purpose was: the more you explore, the more you build your character.

This framework of defining the purpose enabled me to write clearly. I knew how I wanted the audience to perceive my speech. I wrote all the paragraphs and transitions between them with my purpose in mind. I enjoyed writing this speech.

Takeaways

Posture

My posture was good throughout the speech. I had a natural presence and felt confident.

Unlike during the first speech, my neck was straight. Practising in front of the mirror probably helped there. This time around, I observed that my shoulders were not level. It is a tiny thing and may not be noticeable to the audience. I will work on this during my next practice.

I was also tiptoeing during many parts of the speech. My head bobbed up and down a few times. It was unnecessary.

One feedback I got from the evaluator was that I looked away from the camera multiple times. It was not distracting, but it was noticeable. It coincided with the parts where I could not remember the content. It also happened during the last speech. The only solution is to practice more.

The second feedback was a lack of hand gestures. I was more focused on remembering the content that I did not think about gestures. My hands were also not in the camera view. So the audience missed the few natural hand gestures that I had towards the end. So I need to work more on hand gestures.

Setup

I wore formals. I stood up to deliver the speech. The evaluator also mentioned that both the light and the camera angle were good. Notwithstanding, I may have to rethink the setup. My natural hand gestures were not visible because of the camera field view. I will need to increase the distance between the camera and me.

Facial Expression

I did not have a dry throat this time. Thus, there were no weird expressions. I was blank at some places. Otherwise, I was mostly smiling.

Pace

I had the impression that I speak fast. After two speeches, I can say that I have an average pace. The pauses between the sentences and paragraphs are also apt. I do not speak at a snail’s pace, neither is it rushed.

Occasionally, I quickly go through a sentence and mess up. Following are the instances I observed:

• Bystanders sounded like bystandards (or, I may have said bystandards)
• Fixed set of subjects sounded like fix ed of subjects
• Every good thing comes to an end - did not fumble, but it could have been slower.
• I fumbled at regretted while speaking the thing is, I never regretted that decision
• I fumbled at Artificial Intelligence while speaking Data Science and Artificial Intelligence started booming.
• I did not emphasize while ending this sentence: proud of myself after handling this situation without any injuries.

Time

The goal was to take five to seven minutes. Unfortunately, I was way over the time limit. Despite skipping a few sentences, I spoke for nine whopping minutes. It happened because of a lack of insight into my speaking style. If I had timed myself while practising, I would have noticed it. I decided against it because I wanted to practice the delivery and retention of the content.

I wrote the speech assuming a high pace of speaking. It had more than 900 words. I expected to cover all the material in under seven minutes. I could not. Along with the content length, I also missed accounting for pauses between sentences. I will shorten the write up in future speeches. My mentor concurred.

The evaluator suggested starting wrapping up when the yellow timer card comes up. That will push me to cover the relevant parts to finish up on time.

Filler Words

This time around, I used fewer crutch words (28 this time vs 34 last time). The use of uh remained the same, though. :/ I also noticed a new filler: taking a nose breath (as if I had to clear the sinuses).

A Mixture of the Tenses

I mixed the tenses only once. Yay.

I mixed up could, would, and will. Hyperlinked are refreshers on could, would, and should and would vs will.

It looks like I make grammatical mistakes when I am recounting past stories. Practice, practice, practice!!

Opening

The opening of any speech should be grand. It can be a question, a joke, or an eyebrow-raising statement. It will make the audience listen to you.

I created that effect.

The evaluator mentioned that my questions were very engaging. They were engaging because of the following two reasons:

1. Ordering food online is ubiquitous today. Everyone could respond to the questions.
2. The relationship between the questions and the theme was not apparent. That compelled the audience to listen to the next part of my speech.

Here is a little truth about that opening: it was not my first choice.

Initially, I had written a technical opening connecting the Recommendation Systems with the topic. I thought it would grab their attention. Unfortunately, my mentor disagreed and suggested changing it.

Then I wrote a dialogue between me and my friend about how exploration and exploitation emerge while ordering food online. According to my mentor, it would work if I was an experienced speaker. As I am a beginner and I will likely butcher the dialogue delivery. Consequently, I settled on asking questions from the audience.

My mentor hinted that I should think of the opening as a grand event. I will write and present it with more effort if I consider it paramount.

Delivery

Here is the paraphrased version of feedback from my mentor: Initially, I was not sure why I was doing it. Then I gained the confidence and delivered it confidently.

It was my second speech. The environment was very different from the first speech. So I was shaky in the beginning. The audience also surprised me. The replies to my questions were unexpected. I had to make a few changes on the fly.

My mentor says that surprises are inevitable. I should make impromptu changes from the reactions of the audience.

The evaluator commended me on delivering it naturally. It looked like I was comfortable on the virtual stage. My smiling face helped. The more crucial factor was how my speech came across. To my evaluator, we were talking during my speech. Rather than being a performance, it came across as a conversation with the audience. It points in the direction that I connected with my audience. That too on a virtual stage.

My mentor advised maintaining my conversational delivery. At the same time, I should make it impactful. The impact will come if I put more power in my voice. And consistently maintain it. A guiding principle is to make my voice reach the people in the (imaginary) last row.

Content

The evaluator liked my stories. They particularly enjoyed the flow of topics with the theme: food, cycling, school subjects, science, computer science, and data science. The audience found the analogies pleasing. They found the sprinkled humour diverting (special mention for the love triangle bit). The evaluator also mentioned how technical things mixed with the stories made the speech interesting for non-science people. It was surprising to see myself excelling at telling stories because I consider myself mediocre at it.

Lastly, one audience member remarked that she was motivated to explore more. It made me feel that my speech successfully inspired at least one person.

Topic

The speech was about exploring stuff. Exploiting the right ones and further exploring them in depth. So the speech title should have been Explore and Exploit instead of Explore or Exploit.

Speaking vs Writing

My mentor gave me an interesting outlook when I asked him about improving the language of my content and delivering it better.

There is a difference between our writing and speaking patterns. Exceptional speakers have consistent in the way they speak and write. Novice speakers like me write in a manner that will be miles away from my speech.

Written content has finesse because we spend more time on it. My blog posts go through multiple rounds of reviews before I hit publish. So, the first step towards having a quality language is to bring my speaking level up to my writing level. Right now, I go off script quite a lot. I have to do that sensibly and not because I forgot what I wrote.

The next frontier is to improve how I write. My mentor encouraged me to discover my ideal writing. It will give me a sense of the effect I want to achieve with each sentence.

The pertinent question here is, how do we find this impactful writing ideal? I have to expose myself to a variety of authors and written content. It is not limited to books. It can be book reviews or other literary articles. For example, I have subscribed to Literary Hub to get ideas about quality sentence formations. Gradually, I will build a repository of these ideas. Employing them in my writing will take me towards my ideal.

Conclusion

I delivered my second speech. This time my speech was supposed to have a clear purpose. My speech was about the theme of exploration and exploitation in life.

I reproduced all the positive things from last time: camera angle, attire, and expressions. I came across as natural. My pace was right. Pauses between the sentences were natural. I used fewer fillers than the last time. There were a few grammatical mistakes, but nothing major.

My opening was engaging and relatable to the audience. The flow of the speech kept the audience interested. Stories and humour were also well received. I found out that I am good at storytelling. My mentor helped me understand how I can improve the quality of my writing and speeches.

Key improvement areas for me are:

• Time management. I need to finish my content within the stipulated time.
• Practice a lot more
• Bring power in my voice and maintain it throughout the speech.
• Bring speaking level closer to writing level.

My third speech is about vocal variety and body language in my delivery. I am looking forward to more discoveries.

I delivered my first speech on 13th Feb 2021. (Yeah, I procrastinate a lot.)

The first speech is supposed to be an icebreaker. You are just starting, and thus, you first tell your club members about you. It is supposed to be 4 to 6 minutes long. Assuming the pace to be between 120 to 140 words per minute, it comes around 500 to 800 words.

My first draft had a little under 1000 words. My mentor asked me to remove two paragraphs. Thus the final speech had somewhere over 800 words.

This post is about the speech and my takeaways.

Should I Stay, or Should I Go?

I always loved the very first exercise of my school textbooks. They were effortless. I thought my first speech would be the same. Little did I know, Icebreakers are supposed to be about you, and I am always at a loss for words when asked to describe myself. The only time I wrote a bio was on my Tinder profile. I have a suspicion as to why I never got any matches. :)

A few days before Christmas last year, I was oscillating between technical writing and going for a 6-day cycling trip from Dandeli to South Goa. Both had utterly compelling arguments favouring them. After some procrastination and over-thinking, I finally came to my classic quandary: “Should I stay, or should I go?”

And this has been the theme of my life. Today, I wish to show you how my procrastinating, curious, impatient, autodidactic, confident (likely over-confident), risk-taking, and adventurous personality always brings me back to this question.

I was born in MP and grew up in UP and Delhi. A place called Baghpat in UP is my hometown. That is where I learned cycling. I was 9 or 10 years old. Initially, I was just a third wheel between my cousin and his bicycle. I wanted to ride a bike as well. So, I rented the smallest bicycle for a rupee an hour and learned it on my own in 3 days. After that, I had daily trysts with that rental bike.

With time, I got more confident. One day, I thought of riding my elder brother’s bike. The next day, I got up early morning and thought for a few minutes if I should wait for a few months, or should I go and take the risk? Why would it be risky? Because when on the bike, my toes would not reach the floor. I decided to take the risk.

I was maintaining the balance. As I could only pedal in half circles, momentum was slowly building up. Now of course something bad happened. I suddenly hit a speed breaker. And at that moment, I found myself sitting on the frame instead of the saddle, my toes barely touching the road. I knew if I hit the brakes now, I would fall and hurt myself. At some distance, I saw a bunch of hay on the footpath. I manoeuvred the bike to break my fall on that cushion. There were a few laughs from the bystanders. I also laughed with them.

That is how my childhood was. After 5th grade, my family moved to Delhi. That is where I spent all my teens and early twenties. People call that time the most impressionistic time. It was certainly true for me.

[I liked Biology, and because of that, I wanted to become a Doctor. My dad is a PhD in Stats. He and a few of his friends convinced me to take Computer Science (CS) elective in 11th grade. I started liking CS, but I was sad leaving biology behind.] – dropped to reduce the number of words

My favourite subject, physics, led to me being obsessed with Einstein. I learned German to read some German texts authored by him. Unfortunately, reading was not easy even after learning spoken German. But, hey, I can at least say good day and introduce myself in German. :D

During those days, I developed my taste in instrumental music. Newfound interest and inspiration from Einstein made me curious about the violin. Impulsively, I bought the instrument and started learning it myself. Sadly, my impatience made me give it up after a few months. I still have that violin, and I gently weep whenever I see it.

[My love triangle with Physics and Computer Science made it fun when it was time for college. How do you decide between aspiring to work on nuclear physics or astrophysics and solving complex problems through a computer? Reality prevailed, and I opted for CS.] – dropped to reduce the number of words

I started my Computer Science engineering in 2012. Data Science and Artificial Intelligence were booming. The idea that with simple maths applied to data, you can solve problems in any domain was just fascinating to me. In addition to Data Science, I also found a few CS subfields interesting. Ultimately, the question came to decide between software engineering and data science. Its interdisciplinary nature made me go for a career in Data Science.

Being a Data Scientist has given me exposure to many domains. In my 5+ years of work experience, I have applied my skills in the supply chain, fraud consulting, banking, and e-commerce industries. Deciding what to try next has not been easy. Staying at the current place has many perks, but going to the next one also has many exciting opportunities.

I have confronted this question of whether to stay or to go umpteen number of times. My observation is that I have rarely regretted my choice. And whenever it has not been favourable, I could always find my way out. But mostly, it has been diverting.

Just two days before the event, I decided to go on that Dandeli to Goa cycling trip, and it was the most exciting new year I have ever spent. I met a lot of strangers, many of whom have become my comrades now.

Takeaways

Time

It took me 6 minutes and 34 seconds to deliver my speech. That is 34 seconds over the allotted time. This time includes extra time because of the following elements:

• Internet outage for a few seconds
• I fumbled a few times because I could not remember the words.
• Few long pauses because I was dehydrated
• A lot of filler words

Setup

Following is what I did to give the speech a feel of the stage performance:

• I was standing while delivering the speech. It emulated the environment of speaking on a physical stage.
• I wore formals - a shirt and a pant. The audience only saw the shirt. It felt very unnatural delivering the speech in this attire.
• I had put the camera at eye level. If my camera had been lower, it would look like I am looking down upon the audience. They will also be looking up my nose. Having the camera at eye level makes the audience feel like I am looking at them and connecting with them.

This setup corroborated the feedback that I was confident and comfortable with the virtual stage.

Posture

I had a good stage presence because of my prep on the setup. I also came across as feeling natural and confident on the screen. Following are a few observations that I made about my posture:

• My head had a tilt to the side during the first 2-3 minutes. Not sure why I was doing that. The takeaway for me was to practice keeping the posture straight above the shoulders.
• I was not comfortable delivering the speech while standing. I need to practice this further.
• Whenever I could not remember stuff, I looked at the ceiling, avoiding the screen. Of course, I need to remember my content better and practice it many times.

Facial Expression

I was smiling throughout the speech. I feel that it was probably because the content was about me. Nevertheless, smiling gave my delivery a feeling of confidence and candidness. The takeaway is to have a personal element in future speeches so that my smile is natural.

Since the speech was about me, I did not feel a lack of confidence about the content. That probably contributed to how confident I was while speaking.

My facial expression changed at many places:

• Weird appearances at places because I was thirsty and could not speak properly. Of course, hydration is a part of it. But I wonder if it could also be my nervousness. I will get to know more about this in future speeches.
• I was trying to gather saliva to make it easy to speak. I produced unnecessary sounds a few times.
• My lips were not level while speaking. It did not happen much, but it coincided with the head tilt. I will need to practice more.

Pace

The usual pace of speaking is 120 to 140 words per minute. Mine was within the range. I think I can increase it a bit, but that’s not a deal-breaker. There were a few areas of improvement I noticed:

• I unknowingly paused at random places. Probably because of a lack of fluency in my thoughts.
• I messed up the pronunciations of a few words. Most were because I went through them quickly.
  • Suspicion sounded like suspension
  • Technical sounded like pechnical (there is no such word)
  • Quandary sounded like quantry (there is no such word)
  • grew sounded like greeu
  • trysts sounded like treests (similar-sounding, but I should have emphasized the word more)
  • impressionistic - fumbled on this
  • diverting sounded like divertee

Humour

Most of the humorous parts got the response I had expected.

The delivery of the Tinder bit was proper. Most of the audience laughed at it.

The third wheel pun was subtle, but a few members got it. Although the delivery was correct, I messed up at the start.

The gently weep at violin bit made very few members smile.I think the delivery could have been better by my expressions or voice. I am not sure how to do it right now.

The part where I discussed the risk of riding my brother’s bike could also have a better delivery. Although, it may also not have been funny.

Filler Words

I believed that I sparingly used filler words. I was mistaken. I used MANY filler words. Following are the stats:

There were many instances where words like so, like, and, but were not fillers. The above counts do not consider those usages. The first goal is to eliminate the use of filler uh.

Incomplete Sentences

I did this 2-3 times. I started with a sentence that was not how I had written it. Midway, after remembering the exact words, I abruptly began with the original sentence.

It did not harm the message, but it hampered the fluency of the speech.

The solution is to either remember exact words from the speech or be good enough to continue with the paraphrased sentence. I am going to focus on the former for now.

A Mixture of the Tenses

Whenever I recount an episode from the past, I mix the present tense with the past. I miss this when writing blog posts as well. The lack of consistency has the potential to confuse the listeners. It also takes away the effect of your message. Few places where I noticed this happening was:

• Which was my hometown should have been which is my hometown.
• A speed breaker comes should have been a speed breaker came.
• I am hanging on the frame should have been I was hanging on the frame.
• I am raking my brain → I raked my brains.

Awkward English Usage

There were many instances where I made sentences awkward while delivering. There were also grammar mistakes.

• Utterly compelling things could have been utterly compelling arguments.
• I was on saddle on that bike could have been I got on the saddle of that bike, or I got on the saddle.
• I pronounced there’s like theres instead of there is. I feel the latter would have sounded better.
• Same with I’ll. Say, I will.
• I was taking slowly slowly moving my bike. It could have been: my bike was moving forward slowly. Redundant words do not add any new information to the message.
• I slipped from my saddle my feet slipped from my pedal. There are two corrections. It should have been: pedals. And a better phrasing could have been as follows: I slipped from the saddle. My feet also lost the grip off the pedals.
• Data science and AI was booming should have been: Data Science and AI were booming.
• CS fields, which again has a few good applications → CS fields, which again have a few good applications.

Content

The audience found the topic interesting. The chronology was simple to follow: childhood, teenage, and adulthood. Members also enjoyed the stories woven into the theme. In the anonymous feedback, someone gave me a rating of 3/5.

Engaging and interactive for the first attempt, could reduce the number of fillers in upcoming speeches.

This was a great speech and got to know a lot about you and your interest.

Absolutely loved your speech.. a wonderful story woven around an interesting theme that gave us a sneak peak into your journey so far. Looking forward to your future speeches. All the best!

These comments told me that the content was suitable. An experienced member of the club gave me actionable feedback.

I could connect to your cycle incident and to the violin. Do you have more of those defining moments in your life? You should try and string them together. Rather than treat it as an interview where you have to state facts about you. People want to know you and not your resume. I guess the two stories stood out to me. And they way you kind of started recounting them.

You could potentially work on thinking how you can make the audience not just hear it but experience it. That is, by transporting them to the incident. Create the environment by hinting the audience senses. Talk about what they would have seen, what they would have smelt, heard, felt. Then they are not just hearing it but being a part of the experience. Then they will know you a lot more. And it will be extremely powerful. I think you are very capable of doing this. Just try it out.

Essentially, he meant that I should improve my storytelling skills and immerse the audience in the story. Make them experience it vicariously.

Conclusion

The first speech to be delivered in Toastmasters is an icebreaker speech. I had prepared the stage. My speech content was somewhat engaging. According to the feedback, the delivery felt natural and confident.

I found many positives and areas of improvement in my delivery. I have identified the following actionable points:

1. Remember the content well. That will eliminate many language issues.
2. Make the stories more immersive for the audience.
3. Practice delivering my speech by standing in front of a mirror or a camera.

I have also identified the points which I will keep doing more of:

1. Keep smiling. A natural smile comes when the content is personal and relatable to the audience.
2. Maintain the current pace of speaking.
3. Have humour and deliver it well.

My second speech should be a speech with a purpose. In my icebreaker speech, the theme loosely touched with the stories. So, I have to take parts of it and rewrite them to reflect a purpose.

This is the first in a series of two presentations. It was quite a hands-on session, where for each slide, I was showing the changes through terminal. I also used this amazing Git tree visualization tool.

Minutes - History Page

I had to build up the above blank screen to something similar to Nomie’s History page.

Now, I have just started using Flutter and Dart for Minutes. Naturally, I did not know how to build the History page. Since I was not as lucky as I was during the Settings screen creation, I had to work with the online tutorials and build it from scratch. The History page is a list of logs in reverse chronological order. I call each of the log a Record Card.

Record Card Particulars

It is good to list down the broad data points that each Record card should show:

1. The amount of time that has passed from now
2. Creation timestamp
3. Settings for tracker record data - edit, copy, delete, share.
4. Details of the Tracker - emoji, name, value.
5. Note added to the log.
6. People mentioned in the log.
7. Context added to the log note. Context is something that adds metadata labels to the records. Think of it like hashtags.

Nomie’s record card inspired all of these points. I’ll discuss them in detail in the Anatomy section.

The following section talks about layouts in Flutter. Feel free to skip to the anatomy section.

Flutter Layout Concepts

I had to understand all the below topics to build my Record card.

ListTile vs Card

I found many tutorials that used ListTile instead of a Card to build a card type widget. The ListTile classmight have complicated the card design, so I selected the Card class. Besides, the Card class is specifically there to build a card. I also found a few good articles [1, 2] to help me create the card.

Column + Row Combo is all you Need

A Card is a rectangular box having a child widget with the complete card layout. And the card layout is built by Column and Row classes. All the child widgets in a Column are displayed vertically, top to bottom. This order reverses in a Row from left to right. With a combination of these two classes, you can build any complex layout. Here is such an example from Flutter’s documentation on Layouts.

The outer red box is a Row widget that contains a Column (green rectangle) and an image widget. In the image below, this green Column widget becomes a vertical array of two Text and two Row widgets. The two inner Row widgets break into more Row and Column widgets.

Main & Cross Axes

The Row and Column widgets also bring the Main and Cross axes. These axes are convenient ways of aligning child widgets. Even after reading their description from the documentation, I couldn’t figure out their usage. The following figure from Flutter’s documentation on Layouts explains how they align depending on the widget.

Anatomy of the Record Card

I will mark the Record card for every Row and Column block used. I will also go into detail about the displayed information and the code.

Record Card Layout

The Record card has multiple folds of Row and Column widgets. The card itself is a Column widget that contains more blocks.

This Column widget contains five individual Row widgets stacked vertically as below.

The Row widgets from 3 to 5 are simple, non-divisible widgets. The following figure shows the breakage of Row 1 into a Column that further contains two Row widgets.

And Row 2 contains an inner Row that has a single Column.

The blank space after the first Row is for other elements that might come later. I did not want to centre or stretch the card details over the whole Row.

Record Card Particulars - Detailed

Timestamp and Settings popup

The first goal was to have the time passed string like - now, 3 hours ago, 1 day ago, 2 months ago, etc. - from the log creation time. Instead of writing multiple conditional statements, I used the Flutter package called timeago. It gave me precisely the functionality I wanted. Just put in the DateTime object, and it will return the formatted string.

timeago.format(endTime);

The second goal was to display the timestamp. I had to account for two types of Trackers: Counter and Timer. The Counter will have a creation time, and the Timer will have both the start and the end times.

The third objective was to have the three horizontal dots settings popup for each card. The popup menu was easy to create using the PopupMenuButton class. I read various articles [1, 2, 3, 4] to design and position the popup menu on clicking. By default, the PopupMenuButton gives you three vertical dots. Rotating the dots by 90 degrees did not work well. Then I found the icon parameter in the PopupMenuButton class and used the More Horiz icon. Following is the corresponding code:

PopupMenuButton(
    icon: Icon(Icons.more_horiz),
    iconSize: 20,
    itemBuilder: PopupMenu,
);

Mini Tracker Widget

The mini Tracker widget shows the information about the Tracker used to create the tracking log. Each tracker has an emoji and a name. And the log also saves the value of the tracker. For example, in the Record shown above, the mini Tracker’s name is Mood tracker has a Rainbow emoji. And its value in this particular entry was 10 (on a 10-point scale).

The row looks empty because I will have plans to add more information to it once the Minutes’ Tracker page is complete.

Record Note

It is a simple Text widget created to display the note added to the record entry. The names when mentioned using @ should hyperlink to that person’s details. I will add this feature later.

People mentioned in the Record

In the record note, you can add people using the @ prefix. These people will be extracted and saved for later stats. I wanted to highlight the people mentioned in the note.

The Flutter term for the widget displaying each person in the above screenshot is a chip. There are multiple kinds of chips. For the clickable functionality, I used the ActionChip class. This medium article has a good explanation of the chips.

I faced some hiccups in spacing, padding and wrapping of the chip widgets. The linked SO answer showed me how to use padding and labelPadding parameters. I also used the visualDensity parameter. In hindsight, I should have read the documentation first.

I also had to handle the wrapping of the chips to the new line. The Wrap class class exactly does that. The Wrap was not playing well with the Row class. So this answer suggested eliminating the Row class. And the padding between multiline chips was handled using the runSpacing and spacing parameters of the Wrap class. Here is how my code looks:

Wrap(
    runSpacing: -8,
    spacing: 2.0,
    children: List.generate(people.length, (index) {
        return PersonChip(person: people[index]);
    }),
);

Contexts mentioned in the Record

Identical to the last section, I used Wrap to show multiline context chips using all the required padding and spacing parameters.

Other Elements

I used SizedBox class to add blank space between different layout elements. It can add space in both vertical, as well as, horizontal directions.

The Divider class came in handy to add the horizontal line between the last two sections. There are, of course, a host of options available for dividers.

The last thing was to make my card tappable. I wanted to make each record card interactive. Currently, it is just the ripple effect on each tap, but I have thought of opening the record log editing window. I learned that I had to wrap my card widget inside a clickable widget. I had a few options available, out of which I chose the InkWell class.

History Tab

I used the ListView class to display all the Records cards vertically in the History tab.

Lessons Learned

While coding and debugging the errors, I learned many new things.

Flutter/Dart Syntax

Many were syntax related. For example, I need to use x.runtimeType to print the data type of a variable. I also observed the similarity between the if-else conditional statements in Dart and C programming language. The wiki page says that Dart has C-style syntax.

Variable Declaration

Another learning was the difference between the keywords like var, final, and const for declaring a variable. If you explicitly assign a datatype (String, int, List, etc.) to a variable, it becomes static. That means that you can not update its type throughout the runtime. If you do not know the data type beforehand, then you use the var keyword. This way you can assign any data type to the variable throughout the code runtime, just like you can in Python. This variable will be called dynamic. The final keyword indicates that the data type of the variable will not change now. Finally, the const keyword makes the variable immutable. [ref: SO]

DateTime in Flutter

Learning how to manipulate the DateTime class objects in Flutter was also fun. There was no in-built method to convert DateTime variables into formatted strings. I had to install and import the intl package. This package gave me the DateFormat class that had a lot of formatting options available. Python has this whole formatting thing included in the core datetime class.

There was a moment while creating the card where I was puzzled by what was happening. I wanted to display the formatted timestamp on the card, but the timezone was getting mixed up. At some places it was IST, and at the other, it was UTC. To have the same timezone, I went down the timezone rabbit hole. There is no direct support of timezones in Flutter/Dart. But there is a timezones package by Google, but its usage included adding the timezone database as an asset in your app. I wanted to avoid that. I realized that the IST dates already have the timezone info. That means that I have missed giving that information to the remaining dates. Long debugging story short, I had to tell all the DateTime objects to use .toLocal() to pick the timezone information from my device.

Hot Reloading

While building the app, I use hot reloading to see the effects of my changes. There were two issues I have started facing regarding hot reloading. Intermittently, it stops refreshing. First, I try restarting the debug session. If that does not solve the problem, I wipe clean my Android Emulator data and Cold Reboot it.

There was a moment when hot reloading took a lot of time to refresh. The usual time of under 1 second increased to 7 seconds. Running flutter clean solved the problem then, but I am not sure if it is the solution.

Code Organization

The History tab has lead to the creation of a lot of files in my directory. I could not decide between the two possibilities of the code arrangement. There were a lot of articles on this topic. The one blog post I read gave me some clarity, but I will need to read more about it.

Conclusion

The History tab and the Record card for this tab was a decently large task to keep me busy during the weekend. It taught me a lot, and I am satisfied with the final results.

To make the cards look better, I will need to look into card themes once the Minutes v1 is in place. The next post is going to be about either the Tracker or the People screen.

1. History
2. Dash
3. Track
4. People
5. Settings

The screens did not contain any content. Tapping on each icon just switched to that screen (as the gif at the end of the previous post showed).

I had decided to work on the Tracker page as it is the primary function of the app. After reading my previous post, Nikhil wanted to help me out with the app. So we started working on the tracker screen together. He created a simple tracker card. I added on top of it. But soon, we were getting a lot of merge conflicts while rebasing our git branches. So I decided to work on the Settings screen while Nikhil finishes his part on the tracker screen. I will talk about the trackers when we have completed the first draft of it.

In this post, I will discuss the Settings screen.

Minutes - Settings

I had a completely blank Settings screen. Since I was replicating the Settings page of Nomie, I knew various toggles and setting options I wanted to add to Minutes. I googled for a tutorial to get started and instead found Settings UI for Flutter package. This package has abstracted out everything and provided me with SettingsList, SettingsSection, SettingsTile, and CustomSection classes to build the page.

The SettingsList combines various SettingsSection widgets, which further contains many SettingsTile widgets. If there is something you can not do using SettingsSection, you also have CustomSection using which you can build anything your creative mind comes up with.

The documentation for the package is non-existent. There is only the sample flutter app in its Github repo which gets you started. I still had to get into the package code in search of a few options for SettingsTile.

Using the Package

It needs to be added to the pubspec.yaml under the dependencies as follows:

dependencies:
  settings_ui: ^1.0.0

When you save this file, the Dart vscode extension installs it and adds it to the pubspec.lock. After installing the package, I imported it like any other library.

import 'package:settings_ui/settings_ui.dart';

Settings Sections

I followed the sample example to build the working settings page for me and then added all the settings related to Minutes. These settings belong to the following sections:

1. Tracking: Settings related to trackers reside here.
2. Locale: Settings like 24 hr clock or week start day.
3. Security: Enabling fingerprint unlocking or locking app in the background
4. Notifications: Various settings related to notifications
5. Storage: locations where you can store the tracking data - local, Dropbox, Google Drive, Remove server, etc.
6. Import Data: Things related to importing data in Minutes
7. Export Data: How to export data from Minutes
8. Usage Stats: App usage stats. This section might move to the Dash screen later.
9. Danger Zone: Deleting all the data

These nine sections will likely evolve into something else as we will develop the app further.

Currently, I have defined all the sections in a single dart file. I wanted to put each one in a separate file and then use it directly in the main file, but I couldn’t make it work. So that is a to-do for me as a part of code refactoring in future.

Adding Image Assets to the App

In the sample example app from the settings-ui package, there is an image displayed at the footer. I wanted to add it to my Settings page as well. No matter what I did, it was not working. Code always threw an exception as it was not able to find the image. After spending 5 minutes on it, I searched for including image assets in a Flutter app. It turns out I needed to add the image assets to the pubspec.yaml under the assets. Here is how to do it:

  assets:
    - assets/settings.png

Dark/Light Theme

While searching for refactoring the sections code, I found a snippet where the author added the Dark/Light theme support to their app. I hadn’t even looked for it before because I thought it would take a lot of time to add that support. Surprisingly, it was just a short piece of code change that did the whole thing. In the main.dart file, within the MaterialApp, I had to add the following part:

      theme: ThemeData(
        brightness: Brightness.light,
      ),
      darkTheme: ThemeData(
        brightness: Brightness.dark,
      ),

The complete Minutes main app widget looks like this:

class Minutes extends StatelessWidget {
  static const String _title = 'Minutes';

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: _title,
      theme: ThemeData(
        brightness: Brightness.light,
      ),
      darkTheme: ThemeData(
        brightness: Brightness.dark,
      ),
      home: Home(),
    );
  }
}

Conclusion

In quite a short amount of time, the Settings page went from looking all yellow to what you see in the below gif.

Nikhil should be close to completing his part on the tracker page. So next time we should hopefully look at the tracker page.

ActivityWatch is used to track everything you do on your laptop. It has quite an extensive set of features, but it was buggy. So I used it on and off.

Sleep as Android [Play Store] to track my sleep. With the help of this app, I found out how many hours of sleep I require to feel fresh. I still use this app.

I use the My Budget Book] app to track my expenses. It’s only available for the smartphone, so all the tracking happens within the phone. I manually enter all my expenses in this app right when I’ve spent the money. It works for all things except investments. I didn’t find a suitable solution for this. GnuCash seems close, but the account setup and tracking are a bit involved.

Nomie is a simple and quite helpful app. You create different “trackers” for things you want to track and just get started with tracking. Trackers can be count-based, time-based, or text-based. I’ve been using it for almost a year now. It gives a decent view of where I spent my day. It also has a rudimentary dashboard where you can add different graphs based on the trackers you’ve created. Notwithstanding the features, it’s slow, and I couldn’t agree with some of the design decisions of the app.

IFTTT is a tool to collect data from a phone and push it somewhere. I used it to track the time spent on calls and upload the records to Nomie (yes, Nomie also has an API). It used to be free before, but now it’s subscription-based, and their subscription model is costly. It also sometimes refused to work on my device. So I had to give up on it.

I didn’t stick with a lot of other apps because of a multitude of reasons. Some didn’t have data export functionality, meaning my data will always remain locked with them. Others didn’t let me use the app without an account. I am always sceptical about their security and data breaches, so I skipped those. Many others didn’t have cross-platform support. I either have to be stuck on my smartphone or my laptop.

After trying out so many tools, I know what exactly I want to track and how I want the app to look and function. So now, I’ve started working on an app for personal usage. The starting point for it is going to be Nomie. I’ll skip all the things I don’t want. I am going to call this app Minutes.

I am going to use Flutter for this project. Flutter will give me native Android and web apps. That handles the cross-platform support I need for the app. I am going to document my process of building this app. I’ll discuss all the design decisions along with the final output. I’ve never created a smartphone app before. So this blog series is also going to be a documentation of my learnings as well. Flutter uses Dart programming language, so this is also an opportunity to learn Dart. Yay.

Let’s get started with the first part.

Flutter Documentation

The getting started page really gets you started with everything required. It takes time to set up everything as it downloads many things. I did the setup in Linux and macOS. Linux took some figuring out a few path issues, but macOS was straightforward. It helps you set up your IDE as well. There are three available IDEs - Android Studio and IntelliJ, VS Code, and Emacs. I am sure there’d be support for other editors as well. I use VS Code, so everything was smooth for me.

Building the first app was also very effortless because of this well-documented tutorial. The dopamine hit of installing Flutter and creating a hello world app in a single sitting was satisfying.

I understood the basics of Widget, StatelessWidget, and StatefulWidget. While creating the State class corresponding to the StatefulWidget, the class name is prefixed, by default, with an underscore (_). That ensures privacy in Dart. It’s a bit similar to Python, where class methods prefixed with a single underscore are informally considered non-public methods. There’s also a good collection of icons available in Google’s Material design features, which you can directly access using Icons.<icon_id>. There were other things that I didn’t understand and just copied from the tutorial.

Hot reloading is also a cool feature. It hugely reduces the development time.

Minutes Homepage

Let’s discuss the first looks of Minutes. After working through the basics, I started cloning Nomie. I first want to replicate the tracker tab. That is how it looks in Nomie:

The first thing from this page I am going to copy is the bottom navigation. I followed this 2018 article. It almost got me to the final thing. There were a few hiccups.

First, I had to remove the deprecated parts of the code while building each tab in the bottom navigation bar. The original snippet was this:

BottomNavigationBarItem(
    icon: Icon(Icons.home),
    title: Text('Home'),
)

With the help of the flutter dev documentation I converted it to this:

BottomNavigationBarItem(
    icon: Icon(Icons.home),
    label: 'Home',
)

The second problem was much more worrying. When there were three buttons (as were in the blog post), everything worked fine. Though, when I added two more, all the buttons except the active one became invisible. You can see how weird it looks:

I looked around and found a Github issue on the flutter project. The issue was that when more than three BottomNavigationBarItem items are there, and if unspecified, the type of the BottomNavigationBar changes fixed to shifting. This change makes the text and items render in white. One of the comments explains the reasoning behind it. So, it’s a feature, not a bug. Explicitly specifying the type as fixed, fixed the problem:

bottomNavigationBar: BottomNavigationBar(
    type: BottomNavigationBarType.fixed,
    ...
)

And, this is how the Minutes Homepage tabs look right now. When you open the app, the first tab is the “Track” tab. Colours are to add some dummy action on button click.

The next step is to add trackers to the Track tab.

So I decided to create a website having a regularly updated list of all such podcasts. I enjoyed developing this website. This is coming from someone who has never done any decent web development work except tweaking some HTML or CSS here and there. Here’s how I made this free website.

Guiding Principles

I wanted my whole setup to be uncomplicated and quick. Once the design is complete, I don’t want to touch HTML/CSS/JS files again unless for a tweak or update, and even then, it should be minimal. It should be easy to add a new podcast. Most importantly, I wanted most of the things to be automated.

Static Pages

I have no experience with JavaScript so, I skipped any JS Frameworks. I decided to go ahead with a vanilla static site. I used Jekyll to generate static pages that one can host anywhere. I am familiar with Jekyll because I use it to build this blog. Another advantage is that GitHub makes it frictionless to build and host a static website using GitHub pages. Lastly, adding a new blog entry is straightforward: add a simple text file with YAML front matter and boom.

Website Design

I rarely start working on things from scratch. I begin with copying being the first step and then taking it beyond the initial design. Jekyll provides a lot of free themes created by others. I started with Mediumish Jekyll Theme as my first step. It used Bootstrap to design the interface, which was fortunate for me, as I had a tiny bit of experience in it.

I made significant modifications to make the website look the way I had imagined. I knew a large chunk of visitors would visit the website from their mobile devices, so I had to make the website behave properly with small screens, a.k.a. design it responsively. Making that homepage grid behave appropriately with different widths was a bunch of work. Firefox’s Responsive Design Mode helped a lot. With time I figured out all the parts. I learned many new things: Viewport, Flexbox, the background of the Responsive Design from Smashing Magazine, about Media Queries, and more. I also connected with a friend from my previous company for a Flexbox issue I was facing. In the end, I was satisfied with the result.

Designing the website an involved process for me. How do you ensure that it looks how you intend on all devices? Is there something like unit-tests for a front-end or design of a website? I know there are tools like Selenium and others which can open a website in a headless browser and aid in testing its functionality. I found this Stack Overflow question on the topic - What are general tips to test a static website?. Most of the answers were either manual checklists or websites which evaluate your website on few metric.

After I had hosted the website, I evaluated my website through the following links and made a multitude of changes to improve the score:

• WAVE Web Accessibility Evaluation Tool
• Google’s Measure Tool for performance, accessibility, use of best practices, and SEO.

For each podcast, I display a card on the homepage with its image and other details. Minimized the large images with tinypng and GIMP (learned the basics of GIMP). Minified the CSS and JS files, so they take less time to download. Added alt text for images and did most of the suggested accessibility changes. In the process of removing these latencies, I discovered the Coverage Tab in Chrome DevTools. I was using Firefox developer tools, so I didn’t know Chrome had something like this. It was cool.

Hosting

As I mentioned earlier, I used GitHub Pages to host my website. GitHub supports Jekyll right out of the box: I didn’t even have to upload the final HTML files. GitHub will do all of it on its own on every push to the remote repository. One problem was that the hosted website would be available at shivamrana.me/dspods, and I didn’t want the URL. Netlify to the rescue. Netlify hosts a static website and gives you a domain like dspods.netlify.app, where we can put anything in place of dspods. It also directly links with a GitHub repository. Give it build instructions and permission to pull the repository. It’ll pull the code on every push, build, and host it at the specified domain. Of course, you can use your domain name with both GitHub and Netlify.

So Now I’ve automated the continuous deployment pipeline. Update the code, and push it to the GitHub repository. GitHub will build it and host it at the shivamrana.me/dspods URL. Netlify will also build it and host it at the dspods.netlify.app. GitHub build is redundant, but I don’t know of a way to stop it. Thus, the whole pipeline is fully automated. Pretty neat, huh?

Regular Podcast Updates

The only thing remaining is the daily podcast updates. I can’t manually update and push the changes daily. It needs to be automated. GitHub Actions FTW. It was a complete game-changer. Without this, I couldn’t have made this website free.

How does it work? Create a workflow with the instructions to get the updates, commit them, and push them to the repository. A Python script parses the RSS feed of each podcast for updates. The workflow runs daily at midnight. Once the commit is pushed, GitHub and Netlify then get on with their jobs to build and deploy the website.

The whole pipeline is automated now.

Analytics

I’ve also added Google Analytics to see the kind of response there is for the website. You add a small snippet to your website, and it handles all the analytics of your website.

Reception and What’s Next?

When I posted the link on r/MachineLearning, people liked the website and gave me a lot more podcasts to add. You can read the discussion here.

The next thing I want to do is to make the podcast page a bit more informative. I want to provide some insights about the podcasts to the users. The users should understand what kind of things the podcast has discussed in the past.

I also want to work on the discoverability of the podcasts on the homepage. How can a user narrow down the podcast she may want to listen to? If you have any suggestions regarding this, I am happy to hear them.

My hole for this website is to become a go-to place where people find the next Data Science podcast they want to listen to.

Conclusion

I wanted to write about how easy it is to build a free website today. This post was my attempt at that. I hope this will be useful to you. Here’s a laundry list of the tools I’ve used to build this website:

• Free static site generator: Jekyll
• Free hosting: Github Pages and Netlify
• Free domain: Netlify
• Free website evaluation: WAVE Web Accessibility Evaluation Tool, Google’s Measure Tool
• Free website automates updates: GitHub Actions
• Free Analytics: Google Analytics

The open-source website having a collection of podcasts about Machine Learning, Data Science, and ML Engineering is available at - DSPods. The website offers filtering and search options to find podcasts according to your needs. The source code is available at the GitHub Repo. Adding a new podcast is easy: add a simple text file with YAML front matter.

What is Language Identification, you ask?

Code-switching (and code-mixing) is the use of two or more languages in a conversation often employed by multilingual users in informal media like personal chats, Twitter, Facebook, Reddit. Language identification in the code-mixed text is the process of labeling each word with the language it belongs to. For example, in a Hinglish text (code-switching between Hindi and English), a sentence like Hindi ke liye it makes no sense should be labeled as Hindi\Hi ke\Hi liye\Hi it\En makes\En no\En sense\En.

1. 2013 - Labeling the Languages of Words in Mixed-Language Documents using Weakly Supervised Methods
2. 2013 - Query word labeling and Back Transliteration for Indian Languages: Shared task system description
3. 2014 - Word-level Language Identification using CRF: Code-switching Shared Task Report of MSR India System (*this post*)

Publication

Word-level Language Identification using CRF: Code-switching Shared Task Report of MSR India System

Summary

• The objective is to identify the individual language of each word in a code-mixed text for the following four languages: English-Spanish (En-Es), English-Nepali (En-Ne), English-Mandarin (En-Cn), and Standard Arabic-Arabic (Ar-Ar) Dialects.
• They proposed a CRF-based approach inspired by the performance of CRFs in the previous work (discussed here.)
• The methods developed uses various token-based features which can be easily replicated across languages as they are not language-specific.
• The system relies on annotated data for supervised training, and also lexicon of languages, if available.
• The system achieves accuracy ranging from 80%-95% across the four language pairs.

Key Challenges Addressed

• Easy Replication: The features used for training the model are computed using the tokens themselves as they are based on the token context, language lexicon, other character features, and token n-grams. This simplicity enables the easy replication of the methods across various languages.

Constraints/Assumptions

• Since the two languages present in the documents are known, models are aware of the two languages a priori.
• The system assumes that the annotated training data is available for each language pair. In this work, the shared task organizers provided the annotated tweets.

Dataset

They considered the following four language pairs:

1. English-Spanish
2. English-Nepali
3. English-Mandarin
4. Standard Arabic-Arabic

• Released Data (Training and Testing)

  • Twitter data downloaded using the provided Ruby script. The following table summarizes the number of tweets downloaded for each language.

    

  • Pre-processing

    • They excluded the deleted or private tweets from the training set.
    • They fixed a few space tokenization errors by replacing them with an underscore.
• External Training Data
  • Named Entities
    • English from DBpedia instance types - Agent, Award, Device, Holiday, Language, MeansOfTransportation, Name, PersonFunction, Place, and Work
    • Spanish from DBpedia instance types - Agent, Award, Device, Holiday, Language, MeansOfTransportation, Name, PersonFunction, Place, and Work
  • Word frequency lists from Corpus Portal for Search in Monolingual Corpora
    • Available only for English and Spanish languages.
    • Pre-processing: they removed the words containing special characters and numbers from the list.

Methodology

Character n-grams based Feature Engineering

The character n-gram classifiers are used as features in the final system.

• Objective: Two character n-grams classifiers for each language-pair
• Classes: For each language in the pair, they trained a model with binary class classification into two categories: lang1 and others.
• Training data:
  1. 6000 +ve examples randomly sampled from the training set for lang1.
  2. 6000 -ve samples randomly sampled from both the training set and word lists of multiple languages.

  

• Eval data: Nothing mentioned
• Feature Engineering:
  1. Character unigrams
  2. Character bigrams
  3. Character trigrams
  4. Character 4-grams
  5. Character 5-grams
  6. Full word
• Feature Selection:
  • All the features were selected, as past research showed them to be effective.
• Model Selection:
  • The model considered based on previous research: Maximum Entropy (Logistic Regression)
  • MALLET was used to train all the classifiers.

Final Labeling System

• Objective: Training a language identification system for each language pair.
• classes: For each language pair, the labeling system classifies each token into the following six classes:
  1. lang1
  2. lang2
  3. mixed- tokens with morphemes from both lang1 and lang2.
  4. ne- named entities
  5. ambiguous- a word whose label the model cannot determine with certainty in the given context
  6. others- smileys, punctuations, etc.
• Training Data
  • Released data from the shared task (4 language pairs)
  • Named entities dataset
  • Word frequencies
• Eval data: Released data from the shared task (4 language pairs)

• Feature Engineering: The following table shows the list of features created and finally selected for each language pair.

  

  NA: features not applicable or not available. B/U: Bigram/Unigram feature including, the current token.

• Feature Selection: Authors used the 3-fold cross-validation on released training sets to come up with the optimal features reported in the above table for each language pair.

  1. Using all the features of the previous tokens in the bigram context hurt the performance.
  2. The context feature of the previous three and next three tokens was useful.
  3. For En-Es, the character n-gram classifier feature was useful.
  4. For En-Cn, special character features were useful.
  5. For En-Ne, no particular feature set influenced the classification.
  6. Lexicon features were only available for English and Spanish.
• Model Selection:
  • Based on the previous research in language identification, they considered the CRF++ model.
  • MALLET was used to train all the classifiers.

• Model Evaluation: The evaluation metric is the accuracy in a 3-fold cross-validation on the training sets. The following table gives the final accuracies for various combinations of features.

  

  1. Less accuracy of Ar-Ar system
     • Lexicon features of dialectal Arabic were unavailable.
     • Both dialects use the same script.
     • Character n-gram classifier features reduced the accuracy. The hypothesis is that the dialects may not show a drastic difference in their character n-gram distributions.
  2. The En-Cn dataset had En words written in Roman script and Cn words in Chinese script. The feature CHR0 (is English alphabet word?) modeled this signal present in the script.

• Error Analysis:

  

  1. Named Entities (NEs): The F-score of named entities is much lower than the F-scores of lang1 and lang2. Reasons for the discrepancy are:
     • Lack of accurate NE identification systems.
     • Lexicon features only available for English and Spanish.
     • Informal nature of the sentences - not capitalized or spelled properly.
  2. We can ignore the ambiguous and mixed class errors because of their rarity in the datasets. They don’t contribute much to the accuracy of the system.
  3. Reasons for the lower accuracy of Ar-Ar pair (lang1: Arabic; lang2: Dialectal Arabic):
     • The model was only trained on context and word features (and not on lexicon or character n-grams.)
     • Fewer words of dialectal Arabic in both training and testing: couldn’t train a reliable model.
     • Due to distributional skew, the model learned to label the tokens as lang1 with a high probability.
     • F-score for lang2 is 15.8, and that of lang1 is 94.2%: this shows that the identification of lang2 was full of errors.

       

  4. There was a drop in accuracy for En-Ne pair (lang1: English; lang2: Nepali) in surprise data when compared to training and test data. Possible reasons are:
     • The difference in class distribution
     • Genre/style of two datasets: surprise data contained song titles of Nepali songs. Many words were labeled as lang2 by the system, but gold labels identified them as NEs. This classification error was debatable.
     • Results were obtained on only 1,087 tokens, which can’t be used to make any strong claims or conclusions.