Build a Free Data Collection Form in the Cloud with Flask and Python for your Data Science Projects
Introduction
If you need to collect data for a project, this approach will give you full control over your survey questions, the language, terms, even using images - whatever is the best way to communicate with a group. And you can monitor results in realtime and build whatever analytical layer you want. The data gets stored in a CSV ready for analysis and/or modeling.
Code
from IPython.display import Image
Image(filename='animation.gif', width='80%')
Hello Friends, Let's build a simple, free cloud-based survey using Flask and Python
This is a class I always wanted to do but never came around to it until now and as am the Flask-machine-learning guy, here it is.
There will be occasions where you will be occasions where you will tackle a data science project where there is no data. And without data, there won't be much science behind your data science.
I have used similar approaches and seen plenty of others doing it too - if you are working at the grassroot, gorilla approach level, you will need to build your own data collecting mechanism.
Think getting the opinion of recent refugees, the homeless, the immigrant in their own langauge, etc. It is trivial to do and by building it online it will ensure you get consistent answers, don't have to mess around with collecting data manually, etc.
By having full control over your survey or questionaire, you can experiment with different languages, different terms, even use images, whatever is the best way to communicate with a group.
All you need to do is get them a URL and let them answer. I'll show you a fully responsive/mobile first so they can answer it on their phone. We will use W3 templates to get us started and PythonAnywhere to host our app in the cloud and for free.
https://www.pythonanywhere.com/pricing/
To follow along, you will need to get a free PythonAnywhere account and recreate the following structure:
online-survey
├── flask_app.py
├── surveys
└── survey_samp_1.csv
├── templates
└── survey.html
└── static
└── images
└── globe.png
flask_app.py¶
from flask import Flask, session, app, render_template, request, Markup
import sys, io, re
import os, base64
from io import StringIO
from datetime import datetime
import time
app = Flask(__name__)
# get root path for account in cloud
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
# survey page
@app.route("/", methods=['POST', 'GET'])
def survey_page():
message = ''
first_name = ''
last_name = ''
email = ''
gender = ''
are_you_happy = 'Choose one...'
tell_us_more = ''
Family_checked = ''
Friends_checked = ''
Colleagues_checked = ''
# this is a list so create a string to append into csv file
recommend_this_to_string = ''
if request.method == 'POST':
# check that we have all the required fields to append to file
are_you_happy = request.form['are_you_happy']
recommend_this_to = request.form.getlist('recommend_this_to')
tell_us_more = request.form['tell_us_more']
# remove special characters from input for security
tell_us_more = re.sub(r"[^a-zA-Z0-9]","",tell_us_more)
first_name = request.form['first_name']
last_name = request.form['last_name']
email = request.form['email']
date_of_birth = request.form['date_of_birth']
# optional fields
if date_of_birth=='':
date_of_birth = 'NA'
if 'gender' in request.form:
gender = request.form['gender']
else:
gender = 'NA'
# check that essential fields have been filled
message = ''
missing_required_answers_list = []
if are_you_happy == 'Choose one...':
missing_required_answers_list.append('Are you happy?')
if len(recommend_this_to) == 0:
missing_required_answers_list.append('Who would you recommend this survey to?')
else:
for val in recommend_this_to:
recommend_this_to_string += val + ' '
if val == 'Family':
Family_checked = 'checked'
if val == 'Friends':
Friends_checked = 'checked'
if val == 'Colleagues':
Colleagues_checked = 'checked'
if tell_us_more == '':
missing_required_answers_list.append('Tells us more')
if first_name == '':
missing_required_answers_list.append('First name')
if last_name == '':
missing_required_answers_list.append('Last name')
if email == '':
missing_required_answers_list.append('Email')
if len(missing_required_answers_list) > 0:
# return back a string with missing fields
message = '<div class="w3-row-padding w3-padding-16 w3-center"><H3>You missed the following question(s):</H3><font style="color:red;">'
for ms in missing_required_answers_list:
message += '<BR>' + str(ms)
message += '</font></div>'
else:
# append survey answers to file
# create a unique timestamp for this entry
entry_time = datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')
# save to file and send thank you note
with open(BASE_DIR + '/surveys/survey_samp_1.csv','a+') as myfile: # use a+ to append and create file if it doesn't exist
myfile.write(
str(entry_time) + ',' +
str(last_name) + ',' +
str(email) + ',' +
str(date_of_birth) + ',' +
str(are_you_happy) + ',' +
str(recommend_this_to_string) + ',' +
str(tell_us_more) + ','
+ '\n')
# return thank-you message
message = '<div class="w3-row-padding w3-padding-16 w3-center"><H2><font style="color:blue;">Thank you for taking the time to complete this survey</font></H2></div>'
return render_template('survey.html',
message = Markup(message),
first_name = first_name,
last_name = last_name,
email = email,
gender = gender,
tell_us_more = tell_us_more,
Family_checked = Family_checked,
Friends_checked = Friends_checked,
Colleagues_checked = Colleagues_checked,
are_you_happy = are_you_happy)
# used only in local mode
if __name__=='__main__':
app.run(debug=True)
survey.html¶
<!DOCTYPE html>
<html lang="en" >
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Raleway">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<title>Your Survey</title>
<style>
html,body,h1,h2,h3,h4,h5 {font-family: "Raleway", sans-serif}
</style>
</head>
<body>
<!-- Navbar -->
<div class="w3-bar w3-blue w3-card w3-left-align w3-large">
<a class="w3-bar-item w3-button w3-hide-medium w3-hide-large w3-right w3-padding-large w3-hover-white w3-large w3-blue" href="javascript:void(0);" onclick="myFunction()" title="Toggle Navigation Menu"><i class="fa fa-bars"></i></a>
<a href="c{ url_for(survey_page) }}" class="w3-bar-item w3-button w3-padding-large w3-blue">Home</a>
<a href="#" class="w3-bar-item w3-button w3-padding-large w3-blue">About Us</a>
<a href="#" class="w3-bar-item w3-button w3-padding-large w3-blue">Your Rights</a>
</div>
<div class="w3-padding-large">
<div class="w3-row-padding w3-padding-16 w3-center">
<div class="w3-third w3-left-align w3-padding-16">
<img src="/static/images/globe.png" width='100px'/>
</div>
<div class=w3-two-third w3-left-align w3-padding-16">
<h1 class="w3-text-grey w3-padding-16">Survey About Critical Issues in Your Community
<BR><font style="color:#27AE60; font-size: small;">Information your submit will not be shared to outside parties</font></h1>
</div>
{{message}}
<form action="{{ url_for(survey_page) }}" id="survey_page" method="POST">
<H3>Tell us about your Community:</H3>
<div class="w3-row-padding w3-padding-16 w3-center">
<div class="w3-half w3-left-align w3-padding-16">
<label for="are_you_happy"><b><font style="color:#27AE60; font-size: large;">Are you happy?</font></b></label><br>
<SELECT class="w3-button w3-blue w3-padding-large w3-large w3-margin-top" name="are_you_happy">
<option value="{{are_you_happy}}" selected>{{are_you_happy}}</option> ">
<option value="very_happy">Very</option>
<option value="Somewhat_happy">Somewhat</option>
<option value="Neutral_happy">Not sure</option>
<option value="Not_very_happy">Not very</option>
<option value="Not_happy_at_all">Not at all</option>
</SELECT>
</div>
<div class="w3-half w3-left-align w3-padding-16">
<label for="recommend_this_to"><b><font style="color:#27AE60; font-size: large;">Who would you recommend <br>this survey to (multiple answers OK)?</font></b><BR>
<input type="checkbox" name='recommend_this_to' value='Family' {{Family_checked}}>
<label for="Family"> Your family</label><br>
<input type="checkbox" name='recommend_this_to' value="Friends" {{Friends_checked}}>
<label for="Friends"> Your friends</label><br>
<input type="checkbox" name='recommend_this_to' value="Colleagues" {{Colleagues_checked}}>
<label for="Colleagues"> Your colleagues</label><br><br>
</div>
</div>
<div class="w3-row-padding w3-padding-16 w3-center">
<div class="w3-padding-large">
<label for="text"><b><font style="color:#27AE60; font-size: large;">Tells us more:</font></b></label><BR>
<textarea name="tell_us_more" rows="4" cols="30">{{tell_us_more}}
</textarea>
</div>
</div>
<H3>About you:</H3>
<div class="w3-row-padding w3-padding-16 w3-center">
<div class="w3-half w3-left-align w3-padding-16">
<label for="first_name"><b><font style="color:#27AE60; font-size: large;">First name:</font></b><label><br>
<input type="text" name="first_name" value="{{first_name}}" required>
</div>
<div class="w3-half w3-left-align">
<label for="last_name"><b><font style="color:#27AE60; font-size: large;">Last name:</font></b></label><br>
<input type="text" name="last_name" value="{{last_name}}" required>
</div>
</div>
<div class="w3-row-padding w3-padding-16 w3-center">
<div class="w3-half w3-left-align w3-padding-16">
<label for="email"><b><font style="color:#27AE60; font-size: large;">Email:</font></b></label><br>
<input type="email" name="email" value="{{email}}">
</div>
<div class="w3-half w3-left-align w3-padding-16">
<label for="date_of_birth"><b><font style="color:#27AE60; font-size: large;">Date of birth:</font></b></label><br>
<input type="date" name="date_of_birth">
</div>
</div>
<div class="w3-row-padding w3-padding-16 w3-center">
<div class="w3-half w3-left-align w3-padding-16">
<label for="gender"><b><font style="color:#27AE60; font-size: large;">Enter your gender:</font></b></label><br>
<input type="radio" name="gender" value="female"> Female</input><br>
<input type="radio" name="gender" value="male"> Male</input><br>
<input type="radio" name="gender" value="other"> Other</input>
</div>
</div>
<div class="w3-row-padding w3-padding-16 w3-center">
<input type="submit" id="submitButton" value="Submit your survey" name="submit"
class="w3-button w3-blue w3-padding-large w3-large w3-margin-top">
</div>
</form>
</div>
<footer class="w3-container w3-teal w3-center w3-margin-top">
<p>Powered by <a href="https://www.w3schools.com/w3css/default.asp" target="_blank">w3.css</a></p>
</footer>
</body>
</html>
Image(filename='static/images/globe.png', width='20%')
Show Notes
(pardon typos and formatting -these are the notes I use to make the videos)
If you need to collect data for a project, this approach will give you full control over your survey questions, the language, terms, even using images - whatever is the best way to communicate with a group. And you can monitor results in realtime and build whatever analytical layer you want. The data gets stored in a CSV ready for analysis and/or modeling.