Systematisation of Catalogue Integration

Systematisation of Catalogue Integration ABSTRACT: Catalogue integration is a very essential and difficult job which needs to be carried out by various commercial portals and commerce search engines to built and consolidate the products gathered from various data providers. In this paper we discuss about an automated mechanism for integrating products from various providers and thus the process is considered from both the views of portal catalogue and the vendor providing catalogue. This commercial portal has its own taxonomy for all the products which is named as the master taxonomy and the data provider organizes its products in a different taxonomy called as the taxonomy provider. This methodology is based on the taxonomy-aware processing step that adjusts the results of a text based classification so that the products that are similar to the provider’s catalogue will appear closer in the master catalogue. To the best of our knowledge , this is the first unique approach that uses structure of taxonomies in order to upgrade the catalogue integration. The  proposed algorithm is scalable and can be applied to large data-sets in web. The algorithm is implemented on real- world data and has greater accuracy as it takes into account that the relationships between the product categories. INTRODUCTION: The internet is no longer an academic and research oriented network but it is an open book with endless commercial opportunities. Online shopping has increased in recent times. These shopping websites have separate portals which can manage the sellers of the site where products arrive from multiple sellers. This includes e-commerce search engines such as Flipkart,Amazon , Sanpdeal and many common commercial search engines such as google product search and Bing shopping. Each seller will have a separate catalogue for the product available. Internet marketplace are now faced with new challenges that arises from the need to shamelessly integrate enormous number of product catalogue from different sources. Product categorization is the main task that needs to be carried out in here. Hence, we need some mechanism which combines both the approaches, provides accurate classification of products and also scalable to large volume of dataset that is typical on the web. EXISTING SYSTEM: In the existing system there is a difficulty for the customer or the provider to update the details of a product. The ecommerce website has a master catalogue according to which the provider has to upload his/her catalogue of a product. If this catalogue does not match with the master catalogue, then the product will not be accepted by the e-commerce website. This will create a problem for the providers who are un aware of the master catalogue. Not all the providers are aware of these technological details and restrictions. The existing system has imposed more restrictions for the upload of products, which can create a backlog in the business of that shopping website. PROBLEM DEFINITION: Given a source catalogue Ks=(Ps, S,s) that corresponds to some provider’s catalogue defined over the source taxonomy S=(Cs, Es), and a target (or master) catalogue taxonomy Kt = (Pt,T, t) that corresponds to the catalogue of the commercial portal defined over the target (master) taxonomy T = (Ct, Et). The goal is to learn a cross-catalogue labelling function function l: Ps -> Ct that maps products of the source catalogue to the categories of the target catalogue taxonomy. PROJECT SCOPE: Here it makes use of provider taxonomy information to categorize products coming from data providers to the master taxonomy. This approach is based on taxonomy-aware processing step that adjusts the results of a text-based classifier so that the products that are in near-by categories in the provider taxonomy will be assigned near-by categories in the master taxonomy. TACI is scalable to large volume of datasets and it has linear running time with respect to number of input products. It exploits the full structure of the taxonomy, defining relationships between items that belong to different categories, based on the relationship of the categories in the taxonomy tree. TACI provides accurate results when compared other existing approaches SYSTEM DESIGN: This chapter describes the features and modular design of the proposed algorithm. Unlike existing approaches the proposed algorithm explores the entire taxonomy of the provider and master catalogs to find out a classification of products from providers catalog in master taxonomy. The formulation of the taxonomy-aware catalog integration problem is based on structured prediction problem. The optimized classification of products is achieved by designing the algorithm using metric labelling approach. Thus the proposed algorithm involves calculating two measures: Cost indicating assignment of products to categories Cost indicating strength of relationships among product categories Since the existing approaches considered categories as flat collection of classes, pair-wise relationships are considered and hence suffered scalability issues. The proposed algorithm exploits the taxonomy structure to find out the relationship among the categories and uses them to prune the search space thus making the algorithm scalable. Thus the algorithm has linear running time with respect to the input data and is applicable to larger datasets. FUNCTIONAL STEPS: The Taxonomy Aware Catalog Integration is a 2 step process. Base Classification Step: This step does not consider taxonomy structure and utilizes general text classifier. Finds Assignment Cost. Taxonomy-Aware Processing Step: This step involves exploiting the taxonomy structure of both source and target catalogs. Finds Separation Cost. MODULAR DESIGN: PSEUDO CODE: Input: Source catalog Ks, Target taxonomy T, base classifier b, and parameters à ¯Ã‚ Ã‚ ±,k, and ÃŽ ³ Output: A labeling vector à ¢Ã¢â‚¬Å¾Ã¢â‚¬Å" Fs for all x ц Ps do Ï„*Ï„ ц Ct PÏ„b[Ï„|x] if PÏ„b[Ï„*|x] >= à ¯Ã‚ Ã‚ ± then à ¢Ã¢â‚¬Å¾Ã¢â‚¬Å"x Fà ¯Ã‚ Ã‚ ±Ãƒ ¯Ã‚ Ã‚ ± U {x} else Oà ¯Ã‚ Ã‚ ± à ¯Ã‚ Ã‚ ± U {x} Compute TOPk{x} Compute candidate pairs H à ¯Ã‚ Ã‚ ±,k Initialize hash table ÃŽ ¨ to empty for all (ÏÆ', Ï„) â‚ ¬ H à ¯Ã‚ Ã‚ ±,k do ÃŽ ¨[(ÏÆ', Ï„)]=h (ÏÆ', Ï„) for all x ц O à ¯Ã‚ Ã‚ ± do à ¢Ã¢â‚¬Å¾Ã¢â‚¬Å"x Ï„ ц TOP k ( x ) { (1- ÃŽ ³) A COST(x,Ï„)+ ÃŽ ³ ÃŽ ¨[ (sx, Ï„)] } BASE CLASSIFICATION STEPS This step does not consider the structure of both provider and master taxonomies. It uses Naive Bayes text classification result and à ¯Ã‚ Ã‚ ± value to distinguish between fixed and open products. The fixed products are the set of products in the provider catalog with probability of categories greater than the threshold value and thus, Naive Bayes result is taken as the correct category. TAXONOMY AWARE PROCESSING STEP This step involves exploiting the taxonomy structure to find out the relationships among the categories.This needs to find the similarity measure between two categories in both source and target taxonomies. Pair of products (x,y) assigned to category pair having greater similarity measure in provider taxonomy should be assigned to category pair having greater similarity measure in master taxonomy which is ensured by the penalty function (ÃŽ ´). Absolute difference gives the difference between similarity values obtained for a category pair in both source and target taxonomies. Thus this helps in finding out the cost of separating a pair of products which is given by the separation cost. CONCLUSION AND FUTURE WORKS: With the proliferation of data sharing applications that involve multiple data providers the development of automated techniques for catalog integration will be crucial to their success. In this paper, we presented an efficient and scalable automated approach to catalog integration that is based on the use of source category and taxonomy structure information. TACI is a pioneer to catalog integration which exploits the structure of taxonomies to enhance catalog integration with greater accuracy. Here, we have explained with product integration in shopping portals. However, this technique can also be applied to many other important domains which deploy the concept of multiple data sources which contains several categories to be integrated to a single place in a unique way. This includes important verticals such as Local, Travel, Entertainment, etc. This technique was carried out as a supervised learning technique. For future work, we would like to explore semisupervised learning techniques to incrementally retrain the base classifier with elements chosen during the taxonomy-aware calibration step.