Get the "Applied Data Science Edge"!

The ViralML School

Fundamental Market Analysis with Python - Find Your Own Answers On What Is Going on in the Financial Markets

Web Work

Python Web Work - Prototyping Guide for Maker

Use HTML5 Templates, Serve Dynamic Content, Build Machine Learning Web Apps, Grow Audiences & Conquer the World!

Hot off the Press!

The Little Book of Fundamental Market Indicators

My New Book: "The Little Book of Fundamental Analysis: Hands-On Market Analysis with Python" is Out!


Sign up for my newsletter and get my free intro class:

Build a Free Data Collection Form in the Cloud with Flask and Python for your Data Science Projects

Introduction

If you need to collect data for a project, this approach will give you full control over your survey questions, the language, terms, even using images - whatever is the best way to communicate with a group. And you can monitor results in realtime and build whatever analytical layer you want. The data gets stored in a CSV ready for analysis and/or modeling.



If you liked it, please share it:

Code

Let's build a simple, free cloud-based survey with Flask and Python
In [2]:
from IPython.display import Image
Image(filename='animation.gif', width='80%')
Out[2]:
<IPython.core.display.Image object>

Hello Friends, Let's build a simple, free cloud-based survey using Flask and Python

This is a class I always wanted to do but never came around to it until now and as am the Flask-machine-learning guy, here it is.

There will be occasions where you will be occasions where you will tackle a data science project where there is no data. And without data, there won't be much science behind your data science.

I have used similar approaches and seen plenty of others doing it too - if you are working at the grassroot, gorilla approach level, you will need to build your own data collecting mechanism.

Think getting the opinion of recent refugees, the homeless, the immigrant in their own langauge, etc. It is trivial to do and by building it online it will ensure you get consistent answers, don't have to mess around with collecting data manually, etc.

By having full control over your survey or questionaire, you can experiment with different languages, different terms, even use images, whatever is the best way to communicate with a group.

All you need to do is get them a URL and let them answer. I'll show you a fully responsive/mobile first so they can answer it on their phone. We will use W3 templates to get us started and PythonAnywhere to host our app in the cloud and for free.

https://www.pythonanywhere.com/pricing/

https://www.w3schools.com/w3css/w3css_templates.asp

https://www.responsinator.com/

To follow along, you will need to get a free PythonAnywhere account and recreate the following structure:

online-survey
        ├── flask_app.py
        ├── surveys
            └── survey_samp_1.csv
        ├── templates
                └── survey.html
        └── static
                └── images
                    └── globe.png

flask_app.py

In [ ]:
from flask import Flask, session, app, render_template, request, Markup
import sys, io, re
import os, base64
from io import StringIO
from datetime import datetime
import time

app = Flask(__name__)

# get root path for account in cloud
BASE_DIR = os.path.dirname(os.path.abspath(__file__))

# survey page
@app.route("/", methods=['POST', 'GET'])
def survey_page():
    message = ''
    first_name = ''
    last_name = ''
    email = ''
    gender = ''
    are_you_happy = 'Choose one...'
    tell_us_more = ''
    Family_checked = ''
    Friends_checked = ''
    Colleagues_checked = ''
    # this is a list so create a string to append into csv file
    recommend_this_to_string = ''


    if request.method == 'POST':

        # check that we have all the required fields to append to file
        are_you_happy = request.form['are_you_happy']
        recommend_this_to = request.form.getlist('recommend_this_to')
        tell_us_more = request.form['tell_us_more']
        # remove special characters from input for security
        tell_us_more = re.sub(r"[^a-zA-Z0-9]","",tell_us_more)

        first_name = request.form['first_name']
        last_name = request.form['last_name']
        email = request.form['email']
        date_of_birth = request.form['date_of_birth']

        # optional fields
        if date_of_birth=='':
            date_of_birth = 'NA'
        if 'gender' in request.form:
            gender = request.form['gender']
        else:
            gender = 'NA'


        # check that essential fields have been filled
        message = ''
        missing_required_answers_list = []
        if are_you_happy == 'Choose one...':
            missing_required_answers_list.append('Are you happy?')
        if len(recommend_this_to) == 0:
            missing_required_answers_list.append('Who would you recommend this survey to?')
        else:

            for val in recommend_this_to:
                recommend_this_to_string += val + ' '
                if val == 'Family':
                    Family_checked = 'checked'
                if val == 'Friends':
                    Friends_checked = 'checked'
                if val == 'Colleagues':
                    Colleagues_checked = 'checked'

        if tell_us_more == '':
            missing_required_answers_list.append('Tells us more')
        if first_name == '':
            missing_required_answers_list.append('First name')
        if last_name == '':
            missing_required_answers_list.append('Last name')
        if email == '':
            missing_required_answers_list.append('Email')


        if len(missing_required_answers_list) > 0:
            # return back a string with missing fields
            message = '<div class="w3-row-padding w3-padding-16 w3-center"><H3>You missed the following question(s):</H3><font style="color:red;">'
            for ms in missing_required_answers_list:
                message += '<BR>' + str(ms)
            message += '</font></div>'
        else:
            # append survey answers to file

            # create a unique timestamp for this entry
            entry_time = datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')


            # save to file and send thank you note
            with open(BASE_DIR + '/surveys/survey_samp_1.csv','a+') as myfile: # use a+ to append and create file if it doesn't exist
                myfile.write(
                    str(entry_time) + ',' +
                    str(last_name) + ',' +
                    str(email) + ',' +
                    str(date_of_birth) + ',' +
                    str(are_you_happy) + ',' +
                    str(recommend_this_to_string) + ',' +
                    str(tell_us_more) + ','
                    + '\n')

            # return thank-you message
            message = '<div class="w3-row-padding w3-padding-16 w3-center"><H2><font style="color:blue;">Thank you for taking the time to complete this survey</font></H2></div>'


    return render_template('survey.html',
        message = Markup(message),
        first_name = first_name,
        last_name = last_name,
        email = email,
        gender = gender,
        tell_us_more = tell_us_more,
        Family_checked = Family_checked,
        Friends_checked = Friends_checked,
        Colleagues_checked = Colleagues_checked,
        are_you_happy = are_you_happy)


# used only in local mode
if __name__=='__main__':
    app.run(debug=True)

survey.html

In [ ]:
<!DOCTYPE html>
<html lang="en" >
<head>

  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css">
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Raleway">
  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
  <script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>

  <title>Your Survey</title>

  <style>
    html,body,h1,h2,h3,h4,h5 {font-family: "Raleway", sans-serif}
  </style>

</head>
<body>

<!-- Navbar -->

  <div class="w3-bar w3-blue w3-card w3-left-align w3-large">

    <a class="w3-bar-item w3-button w3-hide-medium w3-hide-large w3-right w3-padding-large w3-hover-white w3-large w3-blue" href="javascript:void(0);" onclick="myFunction()" title="Toggle Navigation Menu"><i class="fa fa-bars"></i></a>
    <a href="c{ url_for(survey_page) }}" class="w3-bar-item w3-button w3-padding-large w3-blue">Home</a>
    <a href="#" class="w3-bar-item w3-button w3-padding-large w3-blue">About Us</a>
    <a href="#" class="w3-bar-item w3-button w3-padding-large w3-blue">Your Rights</a>

  </div>


  <div class="w3-padding-large">

 <div class="w3-row-padding w3-padding-16 w3-center">

        <div class="w3-third w3-left-align w3-padding-16">
            <img src="/static/images/globe.png" width='100px'/>
        </div>
        <div class=w3-two-third w3-left-align w3-padding-16">
            <h1 class="w3-text-grey w3-padding-16">Survey About Critical Issues in Your Community
          <BR><font style="color:#27AE60; font-size: small;">Information your submit will not be shared to outside parties</font></h1>
        </div>


  {{message}}

    <form action="{{ url_for(survey_page) }}" id="survey_page" method="POST">

      <H3>Tell us about your Community:</H3>

      <div class="w3-row-padding w3-padding-16 w3-center">
        <div class="w3-half w3-left-align w3-padding-16">
          <label for="are_you_happy"><b><font style="color:#27AE60; font-size: large;">Are you happy?</font></b></label><br>

          <SELECT class="w3-button w3-blue w3-padding-large w3-large w3-margin-top" name="are_you_happy">
            <option value="{{are_you_happy}}" selected>{{are_you_happy}}</option> ">
            <option value="very_happy">Very</option>
            <option value="Somewhat_happy">Somewhat</option>
            <option value="Neutral_happy">Not sure</option>
            <option value="Not_very_happy">Not very</option>
            <option value="Not_happy_at_all">Not at all</option>
          </SELECT>
        </div>

        <div class="w3-half w3-left-align w3-padding-16">
          <label for="recommend_this_to"><b><font style="color:#27AE60; font-size: large;">Who would you recommend <br>this survey to (multiple answers OK)?</font></b><BR>

          <input type="checkbox" name='recommend_this_to' value='Family' {{Family_checked}}>
          <label for="Family"> Your family</label><br>
          <input type="checkbox" name='recommend_this_to' value="Friends" {{Friends_checked}}>
          <label for="Friends"> Your friends</label><br>
          <input type="checkbox" name='recommend_this_to' value="Colleagues" {{Colleagues_checked}}>
          <label for="Colleagues"> Your colleagues</label><br><br>
        </div>
      </div>



      <div class="w3-row-padding w3-padding-16 w3-center">
        <div class="w3-padding-large">
          <label for="text"><b><font style="color:#27AE60; font-size: large;">Tells us more:</font></b></label><BR>
          <textarea name="tell_us_more" rows="4" cols="30">{{tell_us_more}}
          </textarea>
        </div>
      </div>


      <H3>About you:</H3>

      <div class="w3-row-padding w3-padding-16 w3-center">
        <div class="w3-half w3-left-align w3-padding-16">
          <label for="first_name"><b><font style="color:#27AE60; font-size: large;">First name:</font></b><label><br>
          <input type="text" name="first_name" value="{{first_name}}" required>
        </div>

        <div class="w3-half w3-left-align">
          <label for="last_name"><b><font style="color:#27AE60; font-size: large;">Last name:</font></b></label><br>
          <input type="text" name="last_name" value="{{last_name}}" required>
        </div>
      </div>


      <div class="w3-row-padding w3-padding-16 w3-center">
        <div class="w3-half w3-left-align w3-padding-16">
        <label for="email"><b><font style="color:#27AE60; font-size: large;">Email:</font></b></label><br>
        <input type="email" name="email" value="{{email}}">
      </div>

      <div class="w3-half w3-left-align w3-padding-16">
        <label for="date_of_birth"><b><font style="color:#27AE60; font-size: large;">Date of birth:</font></b></label><br>
        <input type="date" name="date_of_birth">
       </div>
      </div>

      <div class="w3-row-padding w3-padding-16 w3-center">
        <div class="w3-half w3-left-align w3-padding-16">
          <label for="gender"><b><font style="color:#27AE60; font-size: large;">Enter your gender:</font></b></label><br>
          <input type="radio" name="gender" value="female"> Female</input><br>
          <input type="radio" name="gender" value="male"> Male</input><br>
          <input type="radio" name="gender" value="other"> Other</input>
        </div>
      </div>


      <div class="w3-row-padding w3-padding-16 w3-center">
        <input type="submit" id="submitButton" value="Submit your survey" name="submit"

        class="w3-button w3-blue w3-padding-large w3-large w3-margin-top">
      </div>


  </form>

</div>


<footer class="w3-container w3-teal w3-center w3-margin-top">

  <p>Powered by <a href="https://www.w3schools.com/w3css/default.asp" target="_blank">w3.css</a></p>
</footer>

</body>
</html>
In [4]:
Image(filename='static/images/globe.png', width='20%')
Out[4]:

Show Notes

(pardon typos and formatting -
these are the notes I use to make the videos)

If you need to collect data for a project, this approach will give you full control over your survey questions, the language, terms, even using images - whatever is the best way to communicate with a group. And you can monitor results in realtime and build whatever analytical layer you want. The data gets stored in a CSV ready for analysis and/or modeling.