Separation of business logic and data access in django

Question

I am writing a project in Django and I see that 80  of the code is in the file models py  This code is confusing and  after a certain time  I cease to understand what is really happening  Here is what bothers me   I find it ugly that my model level  which was supposed to be responsible only for the work with data from a database  is also sending email  walking on API to other services  etc  Also  I find it unacceptable to place business logic in the view  because this way it becomes difficult to control  For example  in my application there are at least three ways to create new instances of User  but technically it should create them uniformly  I do not always notice when the methods and properties of my models become non-deterministic and when they develop side effects   Here is a simple example  At first  the User model was like this  class User db Models        def get present name self           return self name or  Anonymous       def activate self           self status    activated          self save    Over time  it turned into this  class User db Models        def get present name self              property became non-deterministic in terms of database           data is taken from another service by api         return remote api request user name self uid  or  Anonymous        def activate self             method now has a side effect  send message to user          self status    activated          self save           send mail  Your account is activated            self email    What I want is to separate entities in my code   Entities of my database  persistence level  What data does my application keep  Entities of my application  business logic level  What does my application do   What are the good practices to implement such an approach that can be applied in Django

User · Answer

An old question  but I d like to offer my solution anyway  It s based on acceptance that model objects too require some additional functionality while it s awkward to place it within the models py  Heavy business logic may be written separately depending on personal taste  but I at least like the model to do everything related to itself  This solution also supports those who like to have all the logic placed within models themselves   As such  I devised a hack that allows me to separate logic from model definitions and still get all the hinting from my IDE   The advantages should be obvious  but this lists a few that I have observed    DB definitions remain just that - no logic  garbage  attached Model-related logic is all placed neatly in one place All the services  forms  REST  views  have a single access point to logic Best of all  I did not have to rewrite any code once I realised that my models py became too cluttered and had to separate the logic away  The separation is smooth and iterative  I could do a function at a time or entire class or the entire models py    I have been using this with Python 3 4 and greater and Django 1 8 and greater   app models py       from app logic user import UserLogic  class User models Model  UserLogic       field1   models AnyField               field definitions       app logic user py  if False        This allows the IDE to know about the User model and its member fields     from main models import User  class UserLogic object       def logic function self   User                code with hinting working normally       The only thing I can t figure out is how to make my IDE  PyCharm in this case  recognise that UserLogic is actually User model  But since this is obviously a hack  I m quite happy to accept the little nuisance of always specifying type for self parameter

User · Answer

Django is designed to be easely used to deliver web pages  If you are not confortable with this perhaps you should use another solution   I m writting the root or common operations on the model  to have the same interface  and the others on the controller of the model  If I need an operation from other model I import its controller   This approach it s enough for me and the complexity of my applications   Hedde s response is an example that shows the flexibility of django and python itself     Very interesting question anyway

User · Answer

It seems like you are asking about the difference between the data model and the domain model      the latter is where you can find the business logic and entities as perceived by your end user  the former is where you actually store your data  Furthermore  I ve interpreted the 3rd part of your question as  how to notice failure to keep these models separate  These are two very different concepts and it s always hard to keep them separate  However  there are some common patterns and tools that can be used for this purpose  About the Domain Model The first thing you need to recognize is that your domain model is not really about data  it is about actions and questions such as  quot activate this user quot    quot deactivate this user quot    quot which users are currently activated  quot   and  quot what is this user s name  quot   In classical terms  it s about queries and commands  Thinking in Commands Let s start by looking at the commands in your example   quot activate this user quot  and  quot deactivate this user quot   The nice thing about commands is that they can easily be expressed by small given-when-then scenario s   given an inactive user  when the admin activates this user  then the user becomes active  and a confirmation e-mail is sent to the user  and an entry is added to the system log  etc  etc    Such scenario s are useful to see how different parts of your infrastructure can be affected by a single command      in this case your database  some kind of  active  flag   your mail server  your system log  etc  Such scenario s also really help you in setting up a Test Driven Development environment  And finally  thinking in commands really helps you create a task-oriented application  Your users will appreciate this  -  Expressing Commands Django provides two easy ways of expressing commands  they are both valid options and it is not unusual to mix the two approaches  The service layer The service module has already been described by  Hedde  Here you define a separate module and each command is represented as a function  services py def activate user user id       user   User objects get pk user id         set active flag     user active   True     user save          mail user     send mail             etc etc  Using forms The other way is to use a Django Form for each command  I prefer this approach  because it combines multiple closely related aspects   execution of the command  what does it do   validation of the command parameters  can it do this   presentation of the command  how can I do this    forms py class ActivateUserForm forms Form        user id   IntegerField widget   UsernameSelectWidget  verbose name  quot Select a user to activate quot         the username select widget is not a standard Django widget  I just made it up      def clean user id self           user id   self cleaned data  user id           if User objects get pk user id  active              raise ValidationError  quot This user cannot be activated quot             you can also check authorizations etc           return user id      def execute self            quot  quot  quot          This is not a standard method in the forms API  it is intended to replace the           extract-data-from-form-in-view-and-do-stuff  pattern by a more testable pattern            quot  quot  quot          user id   self cleaned data  user id            user   User objects get pk user id             set active flag         user active   True         user save              mail user         send mail                 etc etc  Thinking in Queries You example did not contain any queries  so I took the liberty of making up a few useful queries  I prefer to use the term  quot question quot   but queries is the classical terminology  Interesting queries are   quot What is the name of this user  quot    quot Can this user log in  quot    quot Show me a list of deactivated users quot   and  quot What is the geographical distribution of deactivated users  quot  Before embarking on answering these queries  you should always ask yourself this question  is this   a presentational query just for my templates  and or a business logic query tied to executing my commands  and or a reporting query   Presentational queries are merely made to improve the user interface  The answers to business logic queries directly affect the execution of your commands  Reporting queries are merely for analytical purposes and have looser time constraints  These categories are not mutually exclusive  The other question is   quot do I have complete control over the answers  quot  For example  when querying the user s name  in this context   we do not have any control over the outcome  because we rely on an external API  Making Queries The most basic query in Django is the use of the Manager object  User objects filter active True   Of course  this only works if the data is actually represented in your data model  This is not always the case  In those cases  you can consider the options below  Custom tags and filters The first alternative is useful for queries that are merely presentational  custom tags and template filters  template html  lt h1 gt Welcome     user friendly name    lt  h1 gt   template tags py  register filter def friendly name user       return remote api get cached name user id   Query methods If your query is not merely presentational  you could add queries to your services py  if you are using that   or introduce a queries py module  queries py def inactive users        return User objects filter active False    def users called publysher        for user in User objects all            if remote api get cached name user id      quot publysher quot               yield user   Proxy models Proxy models are very useful in the context of business logic and reporting  You basically define an enhanced subset of your model  You can override a Manager   s base QuerySet by overriding the Manager get queryset   method  models py class InactiveUserManager models Manager       def get queryset self           query set   super InactiveUserManager  self  get queryset           return query set filter active False   class InactiveUser User        quot  quot  quot       gt  gt  gt  for user in InactiveUser objects all                   assert user active is False       quot  quot  quot       objects   InactiveUserManager       class Meta          proxy   True  Query models For queries that are inherently complex  but are executed quite often  there is the possibility of query models  A query model is a form of denormalization where relevant data for a single query is stored in a separate model  The trick of course is to keep the denormalized model in sync with the primary model  Query models can only be used if changes are entirely under your control  models py class InactiveUserDistribution models Model       country   CharField max length 200      inactive user count   IntegerField default 0   The first option is to update these models in your commands  This is very useful if these models are only changed by one or two commands  forms py class ActivateUserForm forms Form         see above         def execute self             see above         query model   InactiveUserDistribution objects get or create country user country          query model inactive user count -  1         query model save    A better option would be to use custom signals  These signals are of course emitted by your commands  Signals have the advantage that you can keep multiple query models in sync with your original model  Furthermore  signal processing can be offloaded to background tasks  using Celery or similar frameworks  signals py user activated   Signal providing args     user    user deactivated   Signal providing args     user     forms py class ActivateUserForm forms Form         see above         def execute self             see above         user activated send robust sender self  user user   models py class InactiveUserDistribution models Model         see above   receiver user activated  def on user activated sender    kwargs           user   kwargs  user           query model   InactiveUserDistribution objects get or create country user country          query model inactive user count -  1         query model save         Keeping it clean When using this approach  it becomes ridiculously easy to determine if your code stays clean  Just follow these guidelines   Does my model contain methods that do more than managing database state  You should extract a command  Does my model contain properties that do not map to database fields  You should extract a query  Does my model reference infrastructure that is not my database  such as mail   You should extract a command   The same goes for views  because views often suffer from the same problem    Does my view actively manage database models  You should extract a command   Some References Django documentation  proxy models Django documentation  signals Architecture  Domain Driven Design

User · Answer

First of all  Don t repeat yourself   Then  please be careful not to overengineer  sometimes it is just a waste of time  and makes someone lose focus on what is important  Review the zen of python from time to time   Take a look at active projects   more people   more need to organize properly the django repository they have a straightforward structure  the pip repository they have a straigtforward directory structure  the fabric repository is also a good one to look at    you can place all your models under yourapp models logicalgroup py  e g User  Group and related models can go under yourapp models users py e g Poll  Question  Answer     could go under yourapp models polls py load what you need in   all   inside of yourapp models   init   py   More about MVC   model is your data   this includes your actual data this also includes your session   cookie   cache   fs   index data  user interacts with controller to manipulate the model   this could be an API  or a view that saves updates your data this can be tuned with request GET   request POST    etc think paging or filtering too   the data updates the view   the templates take the data and format it accordingly APIs even w o templates are part of the view  e g  tastypie or piston this should also account for the middleware     Take advantage of middleware   templatetags   If you need some work to be done for each request  middleware is one way to go    e g  adding timestamps e g  updating metrics about page hits e g  populating a cache  If you have snippets of code that always reoccur for formatting objects  templatetags are good    e g  active tab   url breadcrumbs    Take advantage of model managers   creating User can go in a UserManager models Manager   gory details for instances should go on the models Model  gory details for queryset could go in a models Manager  you might want to create a User one at a time  so you may think that it should live on the model itself  but when creating the object  you probably don t have all the details    Example   class UserManager models Manager      def create user self  username                plain create    def create superuser self  username                may set is superuser field     def activate self  username           may use save   and send mail      def activate in bulk self  queryset           may use queryset update   instead of save           may use send mass mail   instead of send mail     Make use of forms where possible  A lot of boilerplate code can be eliminated if you have forms that map to a model  The ModelForm documentation is pretty good  Separating code for forms from model code can be good if you have a lot of customization  or sometimes avoid cyclic import errors for more advanced uses    Use management commands when possible   e g  yourapp management commands createsuperuser py e g  yourapp management commands activateinbulk py   if you have business logic  you can separate it out   django contrib auth uses backends  just like db has a backend   etc  add a setting for your business logic  e g  AUTHENTICATION BACKENDS  you could use django contrib auth backends RemoteUserBackend you could use yourapp backends remote api RemoteUserBackend you could use yourapp backends memcached RemoteUserBackend delegate the difficult business logic to the backend make sure to set the expectation right on the input output  changing business logic is as simple as changing a setting      backend example   class User db Models       def get present name self              property became not deterministic in terms of database           data is taken from another service by api         return remote api request user name self uid  or  Anonymous     could become   class User db Models      def get present name self         for backend in get backends             try              return backend get present name self           except    make pylint happy              pass       return None   more about design patterns   there s already a good question about design patterns a very good video about practical design patterns django s backends are obvious use of delegation design pattern    more about interface boundaries   Is the code you want to use really part of the models  -  yourapp models Is the code part of business logic  -  yourapp vendor Is the code part of generic tools   libs  -  yourapp libs Is the code part of business logic libs  -  yourapp libs vendor or yourapp vendor libs Here is a good one  can you test your code independently    yes  good    no  you may have an interface problem when there is clear separation  unittest should be a breeze with the use of mocking  Is the separation logical    yes  good    no  you may have trouble testing those logical concepts separately   Do you think you will need to refactor when you get 10x more code    yes  no good  no bueno  refactor could be a lot of work no  that s just awesome     In short  you could have   yourapp core backends py yourapp core models   init   py yourapp core models users py yourapp core models questions py yourapp core backends py yourapp core forms py yourapp core handlers py yourapp core management commands   init   py yourapp core management commands closepolls py yourapp core management commands removeduplicates py yourapp core middleware py yourapp core signals py yourapp core templatetags   init   py yourapp core templatetags polls extras py yourapp core views   init   py yourapp core views users py yourapp core views questions py yourapp core signals py yourapp lib utils py yourapp lib textanalysis py yourapp lib ratings py yourapp vendor backends py yourapp vendor morebusinesslogic py yourapp vendor handlers py yourapp vendor middleware py yourapp vendor signals py yourapp tests test polls py yourapp tests test questions py yourapp tests test duplicates py yourapp tests test ratings py   or anything else that helps you  finding the interfaces you need and the boundaries will help you

User · Answer

I m mostly agree with chosen answer  https   stackoverflow com a 12857584 871392   but want to add option in Making Queries section   One can define QuerySet classes for models for make filter queries and so on  After that you can proxy this queryset class for model s manager  like build-in Manager and QuerySet classes do    Although  if you had to query several data models to get one domain model  it seems more reasonable to me to put this in separate module like suggested before

User · Answer

In Django  MVC structure is as Chris Pratt said  different from classical MVC model used in other frameworks  I think the main reason for doing this is avoiding a too strict application structure  like happens in others MVC frameworks like CakePHP   In Django  MVC was implemented in the following way   View layer is splitted in two  The views should be used only to manage HTTP requests  they are called and respond to them  Views communicate with the rest of your application  forms  modelforms  custom classes  of in simple cases directly with models   To create the interface we use Templates  Templates are string-like to Django  it maps a context into them  and this context was communicated to the view by the application  when view asks    Model layer gives encapsulation  abstraction  validation  intelligence and makes your data object-oriented  they say someday DBMS will also   This doesn t means that you should make huge models py files  in fact a very good advice is to split your models in different files  put them into a folder called  models   make an    init   py  file into this folder where you import all your models and finally use the attribute  app label  of models Model class   Model should abstract you from operating with data  it will make your application simpler  You should also  if required  create external classes  like  tools  for your models You can also use heritage in models  setting the  abstract  attribute of your model s Meta class to  True    Where is the rest  Well  small web applications generally are a sort of an interface to data  in some small program cases using views to query or insert data would be enough  More common cases will use Forms or ModelForms  which are actually  controllers   This is not other than a practical solution to a common problem  and a very fast one  It s what a website use to do   If Forms are not enogh for you  then you should create your own classes to do the magic  a very good example of this is admin application  you can read ModelAmin code  this actually works as a controller  There is not a standard structure  I suggest you to examine existing Django apps  it depends on each case  This is what Django developers intended  you can add xml parser class  an API connector class  add Celery for performing tasks  twisted for a reactor-based application  use only the ORM  make a web service  modify the admin application and more    It s your responsability to make good quality code  respect MVC philosophy or not  make it module based and creating your own abstraction layers  It s very flexible   My advice  read as much code as you can  there are lots of django applications around  but don t take them so seriously  Each case is different  patterns and theory helps  but not always  this is an imprecise cience  django just provide you good tools that you can use to aliviate some pains  like admin interface  web form validation  i18n  observer pattern implementation  all the previously mentioned and others   but good designs come from experienced designers   PS   use  User  class from auth application  from standard django   you can make for example user profiles  or at least read its code  it will be useful for your case

User · Answer

Most comprehensive article on the different options with pros and cons     Idea  1  Fat Models Idea  2  Putting Business Logic in Views Forms Idea  3  Services Idea  4  QuerySets Managers Conclusion   Source  https   sunscrapers com blog where-to-put-business-logic-django

User · Answer

I would have to agree with you  There are a lot of possibilities in django but best place to start is reviewing Django s design philosophy    Calling an API from a model property would not be ideal  it seems like it would make more sense to do something like this in the view and possibly create a service layer to keep things dry  If the call to the API is non-blocking and the call is an expensive one  sending the request to a service worker  a worker that consumes from a queue  might make sense  As per Django s design philosophy models encapsulate every aspect of an  object   So all business logic related to that object should live there       Include all relevant domain logic      Models should encapsulate every aspect of an    object     following Martin Fowler   s Active Record design pattern     The side effects you describe are apparent  the logic here could be better broken down into Querysets and managers  Here is an example   models py  import datetime  from djongo import models from django db models query import QuerySet from django contrib import admin from django db import transaction   class MyUser models Model        present name   models TextField null False  blank True      status   models TextField null False  blank True      last active   models DateTimeField auto now True  editable False         As mentioned you could put this in a template tag to pull it       from cache there  Depending on how it is used  it could be       retrieved from within the admin view or from a custom view       if that is the only place you will use it       def get present name self              property became non-deterministic in terms of database            data is taken from another service by api          return remote api request user name self uid  or  Anonymous         Moved to admin as an action       def activate self               method now has a side effect  send message to user            self status    activated            self save               send email via email service            send mail  Your account is activated            self email        class Meta          ordering     -id      Needed for DRF pagination      def   unicode   self           return      format self pk    class MyUserRegistrationQuerySet QuerySet        def for inactive users self           new date   datetime datetime now   - datetime timedelta days 3 365     3 Years ago         return self filter last active  lte new date year       def by user id self  user ids           return self filter id  in user ids    class MyUserRegistrationManager models Manager        def get query set self           return MyUserRegistrationQuerySet self model  using self  db       def with no activity self           return self get query set   for inactive users     admin py    Then in model admin  class MyUserRegistrationAdmin admin ModelAdmin       actions              send welcome emails              def send activate emails self  request  queryset           rows affected   0         for obj in queryset              with transaction commit on success                      send email  welcome email   request  obj    send email via email service                 obj status    activated                  obj save                   rows affected    1          self message user request   sent  d    rows affected   admin site register MyUser  MyUserRegistrationAdmin

User · Answer

I usually implement a service layer in between views and models  This acts like your project s API and gives you a good helicopter view of what is going on  I inherited this practice from a colleague of mine that uses this layering technique a lot with Java projects  JSF   e g   models py   class Book     author   models ForeignKey User     title   models CharField max length 125      class Meta         app label    library    services py  from library models import Book  def get books limit None    filters           simple service function for retrieving books can be widely extended         return Book objects filter   filters   limit     list  None  will return the entire list   views py  from library services import get books  class BookListView ListView           simple view  e g  implement a  build and  apply filters function         queryset   get books        Mind you  I usually take models  views and services to module level and   separate even further depending on the project s size

User · Answer

Django employs a slightly modified kind of MVC  There s no concept of a  controller  in Django  The closest proxy is a  view   which tends to cause confusion with MVC converts because in MVC a view is more like Django s  template    In Django  a  model  is not merely a database abstraction  In some respects  it shares duty with the Django s  view  as the controller of MVC  It holds the entirety of behavior associated with an instance  If that instance needs to interact with an external API as part of it s behavior  then that s still model code  In fact  models aren t required to interact with the database at all  so you could conceivable have models that entirely exist as an interactive layer to an external API  It s a much more free concept of a  model

[python] Separation of business logic and data access in django

Examples related to python

Examples related to django

Examples related to model-view-controller

Examples related to data-access-layer

Examples related to business-logic-layer